Ways to test web access automagically

[david at thekramers.net (David Kramer)]

On Mon, 10 Jan 2005, Bill Horne wrote:

> Thanks for reading this.
> 
> I'm in need of a way to test that my web server is working,
> automatically, from a location outside my firewall.
> 
> I have Apache running and I use it to show a "test" version of the web
> site for my son's Boy Scout troop, but some of the Scouts have been
> complaining that they can't access it. I've checked out the site, and 
> Apache is running and I can access the pages from my LAN - but when I 
> tried it from an anonymous surfing page, it hangs intermittently.
> 
> Unless Comcast has started to choke port 80 requests, I'm at a loss 
> to explain what's happening, so I'd like to set up a robot to
> periodically test if the server is OK.
> 
> Of course, I'll need a remote machine to run it on, but I have a
> relative with DSL, so I think that's covered. What I need are ideas on
> how best to set up the tests.

I wrote a Python script called servicewatcher.py to do things like this 
and use it for some of my clients.  It's very simple but powerful.  You 
have a config file with commands to run, and if any of the commands return 
a non-zero status, it sends an email to one or more addresses (ie my 
cellphone.).  Note that the commands can be compound, and that's how I 
accomplish complex tests with it.

Ferinstance, to verify that a website is *really* up and not hacked or 
anything, I can do

lynx -dump http://www.example.com/ | grep -q -s 'Text from web page'

which will return non-zero if the website is inaccessible or if that 
content is not found on the page.

Here's another fun one:
ping -c 1 -w 30 example.com | grep -q -s '0% packet loss' || ping -c 1 -w 
30 mit.edu  | grep -q -s '0% packet loss'

which says "Ping example.com, and wait up to 30 seconds for a response.  
If there's any packet loss, then do the same to mit.edu.  If there's
packet loss getting to MIT, then the problem might be on my end, so don't
report it.  But if I can get to MIT, then the problem might be at
example.com, so report it."  Gotta love boolean operators.  Note that the 
single pipes are redirection pipes and the double pipe is a blocking OR, 
so it never pings MIT unless pinging example.com fails.

My script is actually a little more powerful than that, but this is the 
essence of it.  Nearly any watchdog-type task can be reduced to a zero or 
non-zero status with the power of chaining UNIX commands together.

----------------------------------------------------------------------------
DDDD   David Kramer         david at thekramers.net       http://thekramers.net
DK KD  
DKK D  Python is executable pseudocode
DK KD  Perl is executable line noise
DDDD                                                             Bruce Eckel