Kevin Key's Blog

Saturday, November 09, 2013

Weather and Coding Geek Project:  Kevin's Acu-Link to Weather Underground Rapid Fire Updater

I recently ordered and received the Acu-Link Internet Bridge and couldn't wait to set it up. It was very easy to configure with my 5-in-1 (temperature, humidity, rain, wind speed, and wind direction) sensor.  Barometric pressure is reported by the internet bridge.  I then went to Acu-Link.com and could view my own weather report online.  I also installed the Acu-Link Android app.




The next thing I did was set up a free account on Weather Underground. I went back to Acu-Link.com, entered my Weather Underground station ID and password, and began observing my weather data on Weather Underground.  Click the station ID to view my weather station data in real time: KCASANTE15


The sensor transmits data ever 18 seconds. Weather Underground has a feature named Rapid Fire which refreshes real-time weather data every few seconds. I soon discovered, however, that I was only seeing updates on WU every 15 MINUTES.  I thought that something must be wrong, so I searched around the acurite.com forums and found out that Acu-Rite only posts the weather data to Weather Underground every 15 minutes.  How disappointing. People on the forums have been asking for more frequenting refreshing of the data for over a year now.

I knew not to expect Acu-Rite to increase the update frequency anytime soon, so I decided to come up with my own solution: write an app that would scrape the data from AcuLink.com and post it to Weather Underground.


First, I had to scrape the data from Acu-Link's website.  I wrote a VB.NET app that makes use of HTML Agility Pack libraries and good ol' RegEx to scrape the data from AcuLink.com.  Much of the data parsing involves simply scraping the text values from HTML. Wind direction is represented graphically, but fortunately the JPG images are named appropriately. In order to make the wind direction data compatible with Weather Underground, I had to convert the direction (N, S, SW, SSW, etc.) into degrees.  Also, the decimal value for temperature is represented in a different set of tags than the integer value.



In order to calculate rainfall rate (vs. cumulative rainfall for the last 24 hours), I had to save the total rainfall for the past hour - and keep repeating.  I made use of a DataTable for this.  I did something similar for calculating wind gusts (highest wind speed in the past 10 minutes).



Finally, I need to post the data to Weather Underground.  The upload protocol is surprisingly simple.  There's a wiki page that explains it all.

And I made use of a timer control to refresh the data at regular intervals.
So now I have the data posting to Weather Underground every 15 seconds instead of every 15 minutes.  If you'd like a copy of the app, let me know.




2/3/2014: Latest version now includes bug fix that allows you to use alarms on Acu-Link without causing the app to crash.

Download the latest version 2014-02-03 17:30

52 Comments:

Blogger Doug said...

Excellent program. Nice job!

Mon Nov 11, 07:16:00 PM PST  
Blogger Kevin Key said...

Thanks Doug.

Tue Nov 12, 01:43:00 PM PST  
Anonymous Anonymous said...

I am very confused, I am a mac user so dusted off an old laptop, ran windup, which must list ip addresses. I ran wincapp and it says its installed now what. do I plug the ether net into the accurate bridge, then what happens next? On the pc I don't see a program running

Thu Dec 26, 08:22:00 AM PST  
Anonymous Anonymous said...

Great program! Exactly what I was looking for. Thank you very much!!!

Sat Jan 11, 02:45:00 PM PST  
Anonymous John said...

This is a great app Kevin. Thank you!

Sun Jan 12, 09:42:00 PM PST  
Blogger Boguslaw Gaweda said...

Cześć Kevin
Please use the google translator
Przepraszam że pisze w języku polskim, ale nie umiem dobrze pisać po angielsku, a stwierdziłem że ty sobie możesz przetłumaczyć w google, chociaż czasami śmiesznie tłumaczy.
Ale do rzeczy dzięki za ten program /scraping the data from Acu-Link.com/ używam ten program, ponieważ nie mam możliwości mieć włączonego komputera przez 24 godziny.
A tak jak mój komputer nie jest włączony to pracuje bridge Acu-Link, w momencie jak włączam komputer, to włącza mi się twój program, dałem shortcut do katalogu startup.
Pracuje to super.

pozdrawiam z New Jersey
Bogusław KNJWALLI2

Mon Jan 13, 05:22:00 PM PST  
Blogger Boguslaw Gaweda said...

This comment has been removed by the author.

Mon Jan 13, 05:22:00 PM PST  
Blogger Kevin Key said...

Cześć Bogusław Gawęda,

Dzięki za miłe słowa. Cieszę się, że mój program działa dobrze dla Ciebie.

-Kevin

Mon Jan 13, 06:22:00 PM PST  
Blogger Boguslaw Gaweda said...

Hello Kevin,

I have a question for you. I use your program, Scrape. But when I use it on Acu-rite My Backyard Weather, and when an alarm I set up goes off (only for temperature), your program crashes. But when I set an alarm for humidity and it goes off, your programs says the humidity is 0%. Is there a way to fix it so that I can use alarms.

Sincerely,
Boguslaw

Fri Jan 17, 01:18:00 PM PST  
Blogger terry wilson said...

This comment has been removed by a blog administrator.

Fri Jan 17, 09:58:00 PM PST  
Blogger Kevin Key said...

Boguslaw Gaweda,

I will work on a fix for that. Give me a few days.

-Kevin

Fri Jan 17, 10:28:00 PM PST  
Anonymous Anonymous said...

Just went to use the program and instead of seeing 19.1 F i saw 191.0 f. Was there something i forgot to do?

Tue Jan 21, 12:47:00 PM PST  
Blogger Boguslaw Gaweda said...

In the temperature widget, go to Setting and turn on “Display Tenths”.

https://acu-link.com/dashboard

Temperature / setting / Display Tenths

Tue Jan 21, 06:49:00 PM PST  
Anonymous Anonymous said...

Thanks, i thought i did that and went back and apparently it never saved. working now

Thu Jan 23, 03:06:00 PM PST  
Blogger Denny Massie said...

Hey Kevin, thanks but I cant figure out how to install it I click on download and it shows 2 files but it wont do anything, any sugestions?

Mon Jan 27, 02:44:00 PM PST  
Blogger Denny Massie said...

Hey Kevin, thanks but I cant figure out how to install it I click on download and it shows 2 files but it wont do anything, any sugestions?

Mon Jan 27, 02:51:00 PM PST  
Blogger Jay Blizman said...

Kevin,
Attempting to use your set up on a laptop running Win 8.1. Looks like all is good except it can't see the Ethernet port to gather the data. Successfully bridged the eth and wireless. Setup sees about 4 different possible localhost listings. Only one Ethernet port but that doesn't get data. There is activity on the acculink device. Any ideas or thoughts would be appreciated.

Bliz.

Tue Jan 28, 06:59:00 AM PST  
Blogger Jay Blizman said...

Kevin,

Attempted to install your program on Sony 64-bit laptop running Win 8.1
Looked good at first, able to bridge eth port and wireless. But when started just sat and did not collect data. Gave it at least 5 min as requested. No matter which eth port or localhost setting listed in setup drop down. None gathered data.
appeaciate any assistance on this.

Thanks,

Tue Jan 28, 07:02:00 AM PST  
Blogger Kevin Key said...

Hi Jay,

Are the blue lights on the bridge blinking? Also, can you see recent data on Acu-Link.com?

Kevin

Tue Jan 28, 10:14:00 AM PST  
Blogger Jay Blizman said...

Kevin,

Blue lights on bridge blink alternating and stop and stay solid for a short while and then start blinking again.

What does that indicate.

Tue Jan 28, 10:42:00 AM PST  
Blogger Jay Blizman said...

Kevin,

Two thoughts.....

Could the laptop firewall setting be in the way? and/or

Does the patch cable to the bridge have to be a crossover cable?

Bliz

Tue Jan 28, 10:58:00 AM PST  
Blogger Kevin Key said...

Jay,

It sounds like your laptop may require the use of a crossover cable (some require it and others somehow manage to adapt without one). You can get one for around $25 at Radio Shack. You can find one on Amazon for much cheaper - but then you have to wait for it to be delivered.

-Kevin

Tue Jan 28, 11:14:00 AM PST  
Blogger Jay Blizman said...

Thanks Kevin, I will try that next and let you know.

Wed Jan 29, 06:44:00 AM PST  
Blogger Jay Blizman said...

Kevin,

Just wanted to let you know that the crossover cable solved the connection issue.
What determines the update frequency. It seems to vary by a number of seconds.

Jay

Wed Jan 29, 05:09:00 PM PST  
Blogger Jay Blizman said...

Where does the .CSV file get written and what is its name?

Wed Jan 29, 05:10:00 PM PST  
Blogger Kevin Key said...

Jay,

Awesome. The update frequency is approximately every 15-20 seconds.

The .CSV file is written to the same directory in which the app resides.

-Kevin

Wed Jan 29, 05:15:00 PM PST  
Blogger Jay Blizman said...

Forget last question about file location. I found it. Just being stupid.

Wed Jan 29, 05:16:00 PM PST  
Blogger Jay Blizman said...

Kevin,

Just a thought. How about being able to determine where the .CSV file is written? Like on an external HDD with custom names?

Just thinking that would keep the files off the main HDD.

Wed Jan 29, 05:37:00 PM PST  
Blogger PDMGuru said...

Hi Kevin
I love the screen scaper->WU upload
I wondered if you'd considered developing something to push the scraped data to cumulus, that would be really cool since cumulus is excellent and also it does not currently support the acu-rite weather stations

Fri Jan 31, 10:21:00 AM PST  
Blogger Boguslaw Gaweda said...

Hi Kevin,
Do you think you can do anything about the alarms on My Background Weather yet?

Sun Feb 02, 05:42:00 AM PST  
Blogger Kevin Key said...

Jay,

I will look into adding the ability to choose the location of the CSV file real soon.

-Kevin

Mon Feb 03, 07:19:00 PM PST  
Blogger Kevin Key said...

Boguslaw Gaweda,

I'm working on the fix now. Stay tuned.

-Kevin

Mon Feb 03, 07:20:00 PM PST  
Blogger Kevin Key said...

Boguslaw Gaweda,

And now the latest version - with a bug fix for the alarms - is ready for you to download (see link in above blog post).

-Kevin

Mon Feb 03, 07:51:00 PM PST  
Blogger Boguslaw Gaweda said...

now the program works great :)
thanks Kevin

http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KNJWALLI2

Tue Feb 04, 03:28:00 PM PST  
Blogger Kevin Key said...

Boguslaw Gaweda,

Cool. :-)

-Kevin

Tue Feb 04, 08:16:00 PM PST  
Blogger Jason Blanchette said...

Kevin I'd like to know the progress of the app running without having to log in. Thanks.

Thu Feb 13, 07:22:00 AM PST  
Blogger Kevin Key said...

Jason B.,

Hopefully sometime in the near future. I'm busy with my day job and am currently busy with a side job on top of that. Getting the app to run without logging in will involve some major changes.

-Kevin

Fri Feb 14, 12:20:00 AM PST  
Blogger Jason Blanchette said...

I understand.... thank you!

Sun Feb 16, 08:04:00 PM PST  
Blogger Kevin Key said...

Jason:

I found a way to set my app (or any app, for that matter) to run automatically when Windows boots up. Following the instructions here. http://www.sevenforums.com/tutorials/67503-task-create-run-program-startup-log.html The only thing I'd do different is the uncheck the box that says "Stop the task if it runs longer than:".

-Kevin

Fri Feb 21, 11:31:00 PM PST  
Blogger Kevin Key said...

Jay Blizman:

I've added the ability to specify the path and filename for the CSV file. See my other post for the updated version.

-Kevin

Sat Feb 22, 02:24:00 PM PST  
Blogger Boguslaw Gaweda said...

Start / All Programs / Startup / AcuLinkScraper.exe - Shortcut
simply pasting the shortcut

Sat Feb 22, 05:08:00 PM PST  
Anonymous Anonymous said...

the scraper app is telling me it cant login to aculink.com i have checked user name and password they are correct. also says key entry already made.

Tue Mar 04, 08:44:00 PM PST  
Blogger Jay Blizman said...

Kevin,

Thanks for including my requested .CSV location and filename in the latest version. This is working great!

Bliz

Wed Mar 05, 07:22:00 AM PST  
Anonymous Anonymous said...

The scraper app is no longer working can you please help. It has worked so well now says cant log in. Thanks

Wed Mar 12, 12:00:00 PM PDT  
Blogger Joe Spadea said...

Hi Kevin,

I am looking for a way to extract all my weather data since I started using the system 3 months ago - is there a way to do that? I am also a .NET developer.

Joe

Sat Mar 15, 10:15:00 AM PDT  
Blogger Unknown said...

This comment has been removed by the author.

Tue Mar 18, 08:22:00 PM PDT  
Blogger Unknown said...

Hi, Kevin. Is it possible to configure/modify your software so that I could read data from the AcuLink hooked up to the router, as in the AcuRite product stock configuration. Reason is, I'd rather not have to install a second Ethernet card in my PC and set up a bridge, as it is already overloaded. It seems to me that you could read directly from the AcuLink as it should just be another device sitting on the network. Or not?

Tue Mar 18, 08:23:00 PM PDT  
Anonymous rnshagam said...

Hi, Kevin. Is it possible to configure/modify your software so that I could read data from the AcuLink hooked up to the router, as in the AcuRite product stock configuration. Reason is, I'd rather not have to install a second Ethernet card in my PC and set up a bridge, as it is already overloaded. It seems to me that you could read directly from the AcuLink as it should just be another device sitting on the network. Or not?

Tue Mar 18, 08:25:00 PM PDT  
Blogger Kevin Freels said...

Just curious if the login problem on the scraper that mentions a key already in use has been solved by anyone. I would much prefer to do this than to buy an additional network card for my PC and run a bridged internet connection. If this is something that is being addressed then I'll just wait. But if something happened that completely breaks the ability to access the data this way then I'll just have to go that route.

Tue May 13, 11:41:00 AM PDT  
Blogger Kevin Key said...

Hey (Other) Kevin,

I've figured it out: it sounds like you're missing the file HtmlAgilityPack.dll - it should be in the same directory as the app. Put that file in the same directory as the app, restart the app, and you should be good to go. The file is in the ZIP file containing my app.

Tue May 13, 08:42:00 PM PDT  
Anonymous Anonymous said...

Kevin, I had posted the other day about a memory leak issue and being able to retain data such as rain fall amounts if you stop and start the application. I was told you were already aware of these issues and were getting ready to release an update, any news on these items in a future update? Thanks!

Fri May 16, 06:36:00 AM PDT  
Blogger Kevin Key said...

Fix for losing the wind and rain data when the app is restarted is coming any day now.

Mon May 19, 11:41:00 AM PDT  

Post a Comment

Links to this post:

Create a Link

<< Home