[Discuss] Linux project - WX

[richb at pioneer.ci.net (Rich Braun)]

Not long after my last move, the wireless receiver on my antiquated Brookstone
outdoor thermometers broke, reviving my decade-old interest in building a
Linux-based weather observatory.

This is a How-To for the project that I've built since 15-Jan, which you can
view at http://wx.ci.net.  The main thing that's changed since mid-2000s is
that low-power embedded-Linux machines have gotten insanely cheap; alas, the
climate-measuring gear hasn't changed at all.  Oh, and Weather Underground got
bought out by TWC just a couple years ago, alas.  Here goes: I'll share the
assumptions, parts list, and software setup.

Assumptions
-----------
- You're doing this as a home hobbyist, not a professional.

- You have permission/ability to securely mount an anemometer at a high point
near your home, workplace, school or friend's place.

- You want to bear witness to climate changes over time, in a precise way.

- In addition to temperature, you want to measure wind, rainfall, and
barometric pressure

- You want your data fully accessible online or on your mobile device, and to
share it via a website (your own or Weather Underground).

- Once set up, you don't want to have to babysit the hardware.

- You already have a web server somewhere (local or far-away, Apache or nginx:
doesn't matter).

- Optionally, you have a local instance of MySQL available somewhere.

Parts List
----------
- Raspberry Pi with SD Card (type 10, 8GB or bigger) - $30

- Davis Instruments station - choose from
  Vantage Pro 2 wireless with fan-aspirated shield - $695
  Vantage Pro 2 wired - $415
  Vantage Vue wireless - $310
  (Sensors for sunlight UV/visible are bundled at the $850 level)

- 3 batteries (C size) for Davis console - $8

- Davis Instruments logger (required) with WeatherLink - $130

- Dahua 3MP outdoor webcam model IPC-HFW4300S - $125

- PoE injector for webcam - $20 to $40

- 10' steel mounting pole (1-1/4" galvanized natural-gas pipe) with
  end cap U-clamps from Home Depot - $25

Approximate budget: $500 low-end without cam, $1050 high-end with cam

I am not aware of any worthy rival to Davis Instruments; you can get something
that *might* work for less than the low end of my suggested budget, but very
likely will make the whole project a huge, unreliable chore. The Davis
products are woefully obsolete as UI devices, but I swallowed my criticism and
am focused on getting the data online. Dahua does have decent rivals, but at
this price-point there really isn't anything worth considering.

Hardware Setup
--------------
- Find the highest point available, put the end cap on your mounting pole

- Fasten anemometer to top of pole using Davis-included hardware, and the rest
of the instrument array about 3' below the top

- Securely attach pole to a wall or railing at the highest structurally-sound
part of your property. Make sure it doesn't *budge* even in high winds. If you
got a non-wireless version, route the cable to where you place the console.

- Open back of the Davis console, install batteries and the overpriced logger
thingie.  Find your coffee table, take out the Windows software that came with
the logger and put a mug of beer on it.  Sip beer slowly at next step.

- Plug a USB cable from your Rasberry Pi into the logger, and an Ethernet
cable into your LAN (or wifi if you prefer that)

- Figure out your latitude/longitude/elevation, enter them into the station
console

- (Optional) find a good spot for your webcam, run cat5e cable to its location
from your Ethernet switch and test with PoE power injector; verify browser
access with password admin/admin

Software Setup - WX
-------------------
- Get Raspbian onto your SD card; plug a blank card into your Linux or Mac and
use the 'dd' command to copy the Debian Wheezy 2015-01-31 image from
http://www.raspberrypi.org/downloads/ to the card.  Tip for Mac users: use the
"rdisk" device, e.g. /dev/rdisk1, otherwise the copy will take hours.

- Plug an HDMI monitor and keyboard (temporarily) into the Raspberry Pi and
boot it up; go through its dialog, selecting en_US.UTF-8 locale, then log in
as 'pi'.

- Use fdisk/resize2fs (or the Raspbian installer dialog) to increase size of
the root volume (I prefer manual configuration so I can leave some scratch
space available to use in a pinch).

- Invoke: apt-get install xtide xtide-data python-configobj \
       sysstat python-mysqldb python-serial python-usb \
       python-cheetah python-imaging rsync

- Invoke apt-get update, then download weewx_3.0.1-1.deb package from
http://sourceforge.net/projects/weewx/files, install with 'dpkg -i' command

- Verify connectivity with the weather station, and set parameters thus:
   wee_config_device --set-interval 300
   wee_config_device --set-rain-year-start=7 # or whatever month

- Set up a vhost on your webserver.  My apache2 vhost looks like this:

  <VirtualHost *:80>
    # Server Configuration:
    ServerName wx.ci.net
    DocumentRoot "/var/www/htdocs/wx/"
    ServerAdmin richb at instantlinux.net
    ServerSignature email
    DirectoryIndex forecast.html
    HostNameLookups Off
    <Directory /var/www/htdocs/wx>
    	 Options -Indexes FollowSymLinks
      	 AllowOverride All
	 Order allow,deny
	 Allow from all
    </Directory>
  </VirtualHost>

- Set up a username 'wx' and ssh keypair for uploads to the webserver, on both
the Raspberry Pi and the web server: this will be used for rsync. (You can use
an existing username on the webserver if you have to.)

- If you like my WeeGreen 'skin', download it from
https://github.com/instantlinux/weewx-WeeGreen and change the

- Download the forecast extension from MIT:
http://lancet.mit.edu/mwall/projects/weather/

- Invoke: /usr/share/weewx/setup.py install \
       --extension /var/tmp/weewx-forecast-3.0.2.tgz

- The forecast extension comes with its own skin; optionally you can turn that
of if you've decided on WeeGreen (or if you're customizing off mine)

- Register yourself at Weather Underground, get an API key and a station
identifier.  My ID is KCASANFR366.

- If you prefer 12-hour format on your output, change the formats from %H:%M
in /etc/weewx/skins/WeeGreen/skin.conf

- Adjust the other preferences in skin.conf

- In the top-level /etc/weewx/weewx.conf, there are several settings to
update: you can (mostly) ignore the Station section because settings are
overridden by what's set in the console, but you'll want to add your XTide
location and WU api_key to the Forecast section, and station ID/password in
the Wunderground section, rapidfire = True (that part's kinda remarkable), and
your web server user/path info to the Rsync section

- If you have a MySQL server (strongly recommended), make a blank database and
add its credential settings to the Databases section.  You actually need two
separate databases, I call mine weewx_a and weewx_f, because the forecast and
archive engines both write to the same table name.  There's a cool script you
can download to convert from sqlite3 to mysql if you've already got data
collected in sqlite3 (google for 'weewx sqlite3-to-mysql.py').

- Add this line to /etc/fstab:
  tmpfs  /var/www/weewx	tmpfs  nodev,nosuid,noexec,nodiratime,size=4M 0 0

- If you have a central rsyslog (also strongly recommended) create the file
/etc/rsyslog.d/custom.conf with content like the following:
  *.info;*.=notice;*.=warn;auth;user.*;daemon.*;mail.none; \
    cron.!=info;local1.*;local3.none    @@syslog.ci.net:601

Software Setup - Cam
--------------------
Webcam capability is a nice-to-have feature which will drive traffic to your
observatory if your camera has a good vantage point.  (Mine happens to be
aimed at Twin Peaks and the Sutro Tower, with a chunk of western sky where fog
and clouds provide dramatic time-lapse opportunities.) Everywhere has unique
opportunities, so be creative even if all you think you've got is a view of
passing traffic.

The embedded-Linux-based camera I've recommended here, from Dahua, is a
hobbyist's dream compared to the likes of much-hyped cloud-centric products
like Dropcam: from its brower console, you get a whole ton of tunable
settings. It's also going to give you an image 2048x1536 rather than
something-by-720, at a fraction of the price.

- Log into the camera and confirm you can see image in the "Live" tab

- On your local Linux system (could be your Raspberry Pi but I have a
better/faster box for pre-processing images), configure a dedicated ftp
user/password (sorry, no scp: yeah, I know--Dahua is a "security" company).

- In the Dahua's Camera->Video->Snapshot section, select "Best" quality

- In the same section, select Customized interval, and choose how often you
want your time-lapse images to be uploaded (I picked 150 seconds)

- In the Storage->Schedule->Snapshot Schedule, select 24x7 and make sure the
"general" (green) band shows up 7 days a week

- Although I'm using the Dahua strictly as a webcam, note that while you're
installing this, you can build an on-the-cheap burglar alarm out of this thing
by enabling motion-detect video and email alerts in the Event->Video
Detect->Motion Detect section.  Works just like the Dropcam my neighbor showed
me after we got hit by burglars.  Net effect is the burglar's activities are
uploaded to the cloud at least up to the point where he steals your camera.

- OK assuming your camera hasn't gotten stolen in the above step (yes, I took
my own advice, and hope that you're taking your next sip of beer): go to the
Storage->Destination->FTP tab and enter the ftp credentials noted above. I
told it to use a directory called "upload".  The camera will start sending
snapshots to that directory every 150 seconds, in a stupidly-complicated
directory hierarchy.  But because you know bash, you don't care how silly the
pathnames are.

- Once you verify that you're getting the uploads to your local ftp
way-station directory, you can optionally register your web cam at
Wunderground (for inspiration, take a look at any of the Boston-area ones by
browsing maps at wunderground.com, or my SF one at
http://www.wunderground.com/webcams/RichCastroWX/1/show.html); the process
takes only a single "Add Your Webcam" dialog box - pick "ftp" and then tell it
where you are and what you want to name the webcam. (You can have more than
one.) Shrug off your objections to Internet privacy: we, ISIS and the NSA
already know where you are.

- Install ImageMagick and ncftp on your local ftp server and set up a crontab
to run an uploader script.  Notice how my images are a non-standard aspect
ratio: I chose to crop the image, and retain as much image-quality as will fit
in 150Kb, prior to transferring to wunderground.com. Since you objected to
learning bash two steps up, here's my script:

 #! /bin/bash
 # Upload latest image to wunderground

 CAM=$1
 MAX_SIZE=145kb
 MAX_TIME=60
 RETRIES=3

 cd ~/upload/$CAM
 LATEST=`find . -type f -name *.jpg -mmin -5 -print|sort -r |head -1`
 IMG=/tmp/image.jpg
 if [ "$LATEST" != "" ]; then
  convert $LATEST -crop 1920x880+0+0 -define jpeg:extent=$MAX_SIZE $IMG
  while [ $RETRIES -gt 0 ]; do
    START=`date +%s`
    ncftpput -f ~/.ncftp -t $MAX_TIME -V -d /tmp/wxupload-debug.log / $IMG
    RET=$?
    FIN=`date +%s`
    if [ $RET == 0 ]; then
      logger WX upload file=$LATEST bytes=`stat -c %s $IMG` seconds=$((FIN -
START))
      break
    else
      logger -p user.warning WX upload failed file=$LATEST bytes=`stat -c %s
$IMG` seconds=$((FIN - START))
    fi
    RET=$((RET - 1))
    sleep 5
  done
  rm $IMG
 fi

- Your wunderground.com ftp-upload credentials (wunderground is *not* a
security company, so they likewise don't do scp) need to be put into the
upload role user's ~/.ncftp file in this form:

  host webcam.wunderground.com
  user RichCastroWXCAM1
  pass xxxxxx

Wrap-up
-------
Early in the above instructions, I mentioned installing sysstat.  Next step:

- Invoke on the Raspberry Pi: iostat -k -d 60

- Wait 10 minutes and take note of the kB_wrtn values.  You're looking to make
sure the SD card doesn't blow up after a few weeks/months (like mine did last
Friday).

- If you're seeing any activity on the SD card, double-check your rsyslog,
MySQL, and tmpfs mounts.  If you've followed these instructions, the SD card
should get zero activity and the Raspberry Pi will remain reliable for more
years than the external sensors.

- Add monitoring of your webcam(s), Raspberry Pi, and web logs to your
nagios/zabbix instance

- Install the wunderground app on your mobile device

- Make sure you've got a one-time backup of the Raspberry Pi's SD card, and
daily backups of the video upload directory on your ftp server and your mysql
database.

- Enjoy a second mug of Sam Adams

---------------
That's it, hopefully this HOWTO inspires you to bear witness to global climate
change (you don't trust the TV and websites to give you unbiased info, do
you?), and saves you lots of time as you juggle the software and hardware
bits.

My apologies for the snow in Boston at this time, and for the fact that unless
you shell out an extra $kilobuck, the only automated measurement of snowfall
you can make is its water content.

-rich
Footnote: the abbreviation "WX" for weather is familiar to me from work in
aviation software. But it actually comes from the days of telegraph machines:
in Morse code it is .-- -..-