[Discuss] Linux project - WX

[Greg Rundlett (freephile)]

Thanks for sharing that write-up Rich!

I've got a crappy (consumer) wireless weather station in my back yard.  I'd
like to someday upgrade it like the setup you did.  So, I'm keeping a copy
of your notes for future reference.  I put it into my wiki at
https://freephile.org/wiki/index.php/Linux_weather_station  Feel free to
reference that and/or let me know if you'd like a wiki account to add/edit.

Greg Rundlett
http://eQuality-Tech.com
http://freephile.org

On Tue, Feb 10, 2015 at 2:10 AM, Rich Braun <richb at pioneer.ci.net> wrote:

> Not long after my last move, the wireless receiver on my antiquated
> Brookstone
> outdoor thermometers broke, reviving my decade-old interest in building a
> Linux-based weather observatory.
>
> This is a How-To for the project that I've built since 15-Jan, which you
> can
> view at http://wx.ci.net.  The main thing that's changed since mid-2000s
> is
> that low-power embedded-Linux machines have gotten insanely cheap; alas,
> the
> climate-measuring gear hasn't changed at all.  Oh, and Weather Underground
> got
> bought out by TWC just a couple years ago, alas.  Here goes: I'll share the
> assumptions, parts list, and software setup.
>
> Assumptions
> -----------
> - You're doing this as a home hobbyist, not a professional.
>
> - You have permission/ability to securely mount an anemometer at a high
> point
> near your home, workplace, school or friend's place.
>
> - You want to bear witness to climate changes over time, in a precise way.
>
> - In addition to temperature, you want to measure wind, rainfall, and
> barometric pressure
>
> - You want your data fully accessible online or on your mobile device, and
> to
> share it via a website (your own or Weather Underground).
>
> - Once set up, you don't want to have to babysit the hardware.
>
> - You already have a web server somewhere (local or far-away, Apache or
> nginx:
> doesn't matter).
>
> - Optionally, you have a local instance of MySQL available somewhere.
>
> Parts List
> ----------
> - Raspberry Pi with SD Card (type 10, 8GB or bigger) - $30
>
> - Davis Instruments station - choose from
>   Vantage Pro 2 wireless with fan-aspirated shield - $695
>   Vantage Pro 2 wired - $415
>   Vantage Vue wireless - $310
>   (Sensors for sunlight UV/visible are bundled at the $850 level)
>
> - 3 batteries (C size) for Davis console - $8
>
> - Davis Instruments logger (required) with WeatherLink - $130
>
> - Dahua 3MP outdoor webcam model IPC-HFW4300S - $125
>
> - PoE injector for webcam - $20 to $40
>
> - 10' steel mounting pole (1-1/4" galvanized natural-gas pipe) with
>   end cap U-clamps from Home Depot - $25
>
> Approximate budget: $500 low-end without cam, $1050 high-end with cam
>
> I am not aware of any worthy rival to Davis Instruments; you can get
> something
> that *might* work for less than the low end of my suggested budget, but
> very
> likely will make the whole project a huge, unreliable chore. The Davis
> products are woefully obsolete as UI devices, but I swallowed my criticism
> and
> am focused on getting the data online. Dahua does have decent rivals, but
> at
> this price-point there really isn't anything worth considering.
>
> Hardware Setup
> --------------
> - Find the highest point available, put the end cap on your mounting pole
>
> - Fasten anemometer to top of pole using Davis-included hardware, and the
> rest
> of the instrument array about 3' below the top
>
> - Securely attach pole to a wall or railing at the highest
> structurally-sound
> part of your property. Make sure it doesn't *budge* even in high winds. If
> you
> got a non-wireless version, route the cable to where you place the console.
>
> - Open back of the Davis console, install batteries and the overpriced
> logger
> thingie.  Find your coffee table, take out the Windows software that came
> with
> the logger and put a mug of beer on it.  Sip beer slowly at next step.
>
> - Plug a USB cable from your Rasberry Pi into the logger, and an Ethernet
> cable into your LAN (or wifi if you prefer that)
>
> - Figure out your latitude/longitude/elevation, enter them into the station
> console
>
> - (Optional) find a good spot for your webcam, run cat5e cable to its
> location
> from your Ethernet switch and test with PoE power injector; verify browser
> access with password admin/admin
>
> Software Setup - WX
> -------------------
> - Get Raspbian onto your SD card; plug a blank card into your Linux or Mac
> and
> use the 'dd' command to copy the Debian Wheezy 2015-01-31 image from
> http://www.raspberrypi.org/downloads/ to the card.  Tip for Mac users:
> use the
> "rdisk" device, e.g. /dev/rdisk1, otherwise the copy will take hours.
>
> - Plug an HDMI monitor and keyboard (temporarily) into the Raspberry Pi and
> boot it up; go through its dialog, selecting en_US.UTF-8 locale, then log
> in
> as 'pi'.
>
> - Use fdisk/resize2fs (or the Raspbian installer dialog) to increase size
> of
> the root volume (I prefer manual configuration so I can leave some scratch
> space available to use in a pinch).
>
> - Invoke: apt-get install xtide xtide-data python-configobj \
>        sysstat python-mysqldb python-serial python-usb \
>        python-cheetah python-imaging rsync
>
> - Invoke apt-get update, then download weewx_3.0.1-1.deb package from
> http://sourceforge.net/projects/weewx/files, install with 'dpkg -i'
> command
>
> - Verify connectivity with the weather station, and set parameters thus:
>    wee_config_device --set-interval 300
>    wee_config_device --set-rain-year-start=7 # or whatever month
>
> - Set up a vhost on your webserver.  My apache2 vhost looks like this:
>
>   <VirtualHost *:80>
>     # Server Configuration:
>     ServerName wx.ci.net
>     DocumentRoot "/var/www/htdocs/wx/"
>     ServerAdmin richb at instantlinux.net
>     ServerSignature email
>     DirectoryIndex forecast.html
>     HostNameLookups Off
>     <Directory /var/www/htdocs/wx>
>          Options -Indexes FollowSymLinks
>          AllowOverride All
>          Order allow,deny
>          Allow from all
>     </Directory>
>   </VirtualHost>
>
> - Set up a username 'wx' and ssh keypair for uploads to the webserver, on
> both
> the Raspberry Pi and the web server: this will be used for rsync. (You can
> use
> an existing username on the webserver if you have to.)
>
> - If you like my WeeGreen 'skin', download it from
> https://github.com/instantlinux/weewx-WeeGreen and change the
>
> - Download the forecast extension from MIT:
> http://lancet.mit.edu/mwall/projects/weather/
>
> - Invoke: /usr/share/weewx/setup.py install \
>        --extension /var/tmp/weewx-forecast-3.0.2.tgz
>
> - The forecast extension comes with its own skin; optionally you can turn
> that
> of if you've decided on WeeGreen (or if you're customizing off mine)
>
> - Register yourself at Weather Underground, get an API key and a station
> identifier.  My ID is KCASANFR366.
>
> - If you prefer 12-hour format on your output, change the formats from
> %H:%M
> in /etc/weewx/skins/WeeGreen/skin.conf
>
> - Adjust the other preferences in skin.conf
>
> - In the top-level /etc/weewx/weewx.conf, there are several settings to
> update: you can (mostly) ignore the Station section because settings are
> overridden by what's set in the console, but you'll want to add your XTide
> location and WU api_key to the Forecast section, and station ID/password in
> the Wunderground section, rapidfire = True (that part's kinda remarkable),
> and
> your web server user/path info to the Rsync section
>
> - If you have a MySQL server (strongly recommended), make a blank database
> and
> add its credential settings to the Databases section.  You actually need
> two
> separate databases, I call mine weewx_a and weewx_f, because the forecast
> and
> archive engines both write to the same table name.  There's a cool script
> you
> can download to convert from sqlite3 to mysql if you've already got data
> collected in sqlite3 (google for 'weewx sqlite3-to-mysql.py').
>
> - Add this line to /etc/fstab:
>   tmpfs  /var/www/weewx tmpfs  nodev,nosuid,noexec,nodiratime,size=4M 0 0
>
> - If you have a central rsyslog (also strongly recommended) create the file
> /etc/rsyslog.d/custom.conf with content like the following:
>   *.info;*.=notice;*.=warn;auth;user.*;daemon.*;mail.none; \
>     cron.!=info;local1.*;local3.none    @@syslog.ci.net:601
>
> Software Setup - Cam
> --------------------
> Webcam capability is a nice-to-have feature which will drive traffic to
> your
> observatory if your camera has a good vantage point.  (Mine happens to be
> aimed at Twin Peaks and the Sutro Tower, with a chunk of western sky where
> fog
> and clouds provide dramatic time-lapse opportunities.) Everywhere has
> unique
> opportunities, so be creative even if all you think you've got is a view of
> passing traffic.
>
> The embedded-Linux-based camera I've recommended here, from Dahua, is a
> hobbyist's dream compared to the likes of much-hyped cloud-centric products
> like Dropcam: from its brower console, you get a whole ton of tunable
> settings. It's also going to give you an image 2048x1536 rather than
> something-by-720, at a fraction of the price.
>
> - Log into the camera and confirm you can see image in the "Live" tab
>
> - On your local Linux system (could be your Raspberry Pi but I have a
> better/faster box for pre-processing images), configure a dedicated ftp
> user/password (sorry, no scp: yeah, I know--Dahua is a "security" company).
>
> - In the Dahua's Camera->Video->Snapshot section, select "Best" quality
>
> - In the same section, select Customized interval, and choose how often you
> want your time-lapse images to be uploaded (I picked 150 seconds)
>
> - In the Storage->Schedule->Snapshot Schedule, select 24x7 and make sure
> the
> "general" (green) band shows up 7 days a week
>
> - Although I'm using the Dahua strictly as a webcam, note that while you're
> installing this, you can build an on-the-cheap burglar alarm out of this
> thing
> by enabling motion-detect video and email alerts in the Event->Video
> Detect->Motion Detect section.  Works just like the Dropcam my neighbor
> showed
> me after we got hit by burglars.  Net effect is the burglar's activities
> are
> uploaded to the cloud at least up to the point where he steals your camera.
>
> - OK assuming your camera hasn't gotten stolen in the above step (yes, I
> took
> my own advice, and hope that you're taking your next sip of beer): go to
> the
> Storage->Destination->FTP tab and enter the ftp credentials noted above. I
> told it to use a directory called "upload".  The camera will start sending
> snapshots to that directory every 150 seconds, in a stupidly-complicated
> directory hierarchy.  But because you know bash, you don't care how silly
> the
> pathnames are.
>
> - Once you verify that you're getting the uploads to your local ftp
> way-station directory, you can optionally register your web cam at
> Wunderground (for inspiration, take a look at any of the Boston-area ones
> by
> browsing maps at wunderground.com, or my SF one at
> http://www.wunderground.com/webcams/RichCastroWX/1/show.html); the process
> takes only a single "Add Your Webcam" dialog box - pick "ftp" and then
> tell it
> where you are and what you want to name the webcam. (You can have more than
> one.) Shrug off your objections to Internet privacy: we, ISIS and the NSA
> already know where you are.
>
> - Install ImageMagick and ncftp on your local ftp server and set up a
> crontab
> to run an uploader script.  Notice how my images are a non-standard aspect
> ratio: I chose to crop the image, and retain as much image-quality as will
> fit
> in 150Kb, prior to transferring to wunderground.com. Since you objected to
> learning bash two steps up, here's my script:
>
>  #! /bin/bash
>  # Upload latest image to wunderground
>
>  CAM=$1
>  MAX_SIZE=145kb
>  MAX_TIME=60
>  RETRIES=3
>
>  cd ~/upload/$CAM
>  LATEST=`find . -type f -name *.jpg -mmin -5 -print|sort -r |head -1`
>  IMG=/tmp/image.jpg
>  if [ "$LATEST" != "" ]; then
>   convert $LATEST -crop 1920x880+0+0 -define jpeg:extent=$MAX_SIZE $IMG
>   while [ $RETRIES -gt 0 ]; do
>     START=`date +%s`
>     ncftpput -f ~/.ncftp -t $MAX_TIME -V -d /tmp/wxupload-debug.log / $IMG
>     RET=$?
>     FIN=`date +%s`
>     if [ $RET == 0 ]; then
>       logger WX upload file=$LATEST bytes=`stat -c %s $IMG` seconds=$((FIN
> -
> START))
>       break
>     else
>       logger -p user.warning WX upload failed file=$LATEST bytes=`stat -c
> %s
> $IMG` seconds=$((FIN - START))
>     fi
>     RET=$((RET - 1))
>     sleep 5
>   done
>   rm $IMG
>  fi
>
> - Your wunderground.com ftp-upload credentials (wunderground is *not* a
> security company, so they likewise don't do scp) need to be put into the
> upload role user's ~/.ncftp file in this form:
>
>   host webcam.wunderground.com
>   user RichCastroWXCAM1
>   pass xxxxxx
>
> Wrap-up
> -------
> Early in the above instructions, I mentioned installing sysstat.  Next
> step:
>
> - Invoke on the Raspberry Pi: iostat -k -d 60
>
> - Wait 10 minutes and take note of the kB_wrtn values.  You're looking to
> make
> sure the SD card doesn't blow up after a few weeks/months (like mine did
> last
> Friday).
>
> - If you're seeing any activity on the SD card, double-check your rsyslog,
> MySQL, and tmpfs mounts.  If you've followed these instructions, the SD
> card
> should get zero activity and the Raspberry Pi will remain reliable for more
> years than the external sensors.
>
> - Add monitoring of your webcam(s), Raspberry Pi, and web logs to your
> nagios/zabbix instance
>
> - Install the wunderground app on your mobile device
>
> - Make sure you've got a one-time backup of the Raspberry Pi's SD card, and
> daily backups of the video upload directory on your ftp server and your
> mysql
> database.
>
> - Enjoy a second mug of Sam Adams
>
> ---------------
> That's it, hopefully this HOWTO inspires you to bear witness to global
> climate
> change (you don't trust the TV and websites to give you unbiased info, do
> you?), and saves you lots of time as you juggle the software and hardware
> bits.
>
> My apologies for the snow in Boston at this time, and for the fact that
> unless
> you shell out an extra $kilobuck, the only automated measurement of
> snowfall
> you can make is its water content.
>
> -rich
> Footnote: the abbreviation "WX" for weather is familiar to me from work in
> aviation software. But it actually comes from the days of telegraph
> machines:
> in Morse code it is .-- -..-
>
>
> _______________________________________________
> Discuss mailing list
> Discuss at blu.org
> http://lists.blu.org/mailman/listinfo/discuss
>