[Discuss] adventures in dns

[me at mattgillen.net (Matthew Gillen)]

For those that still have power and time to kill...

So the rolling power outages today at my house gave me a chance to swap out
a dead hard drive in my home server that had been sitting there unplugged
but still screwed in (I'd ordered and received a replacement a while back).

This eventually led me to learn a new thing to watch out for when DNS isn't
working as you'd expect.  Note that I'm running my own DNS server for my
home network (primarily a caching-only server, but I also use it to block
some domains that I don't like).

So I get the new drive in, plug everything back in and fire it up.  Get some
error about the CMOS checksum being screwed up, resetting to factory
defaults.  So I go through and make my usual tweaks (most important being
enabling AHCI mode for SATA).  Boot up, and sort of looks like I'm in
business.  Can't ping www.google.com though (lookup failure).  Can ping my
router, and can ping a couple well-known external raw IP addresses.

My server only has one entry in /etc/resolv.conf, pointing to 127.0.0.1.
This is normally the behavior I want; I want my server getting the same
results as every other machine on my network (e.g. blocking the same
domains, etc).  However...

I check named's log file and see a bunch of messages like this:
> 05-Nov-2008 19:17:27.336 error (broken trust chain) resolving './NS/IN': 8.8.8.8#53
> 05-Nov-2008 19:17:27.524 validating @0x7f112c052ea0: . DNSKEY: verify failed due to bad signature (keyid=19036): RRSIG validity period has not begun
> 05-Nov-2008 19:17:27.524 validating @0x7f112c052ea0: . DNSKEY: unable to find a DNSKEY which verifies the DNSKEY RRset and also matches a trusted key for '.'
> 05-Nov-2008 19:17:27.524 validating @0x7f112c052ea0: . DNSKEY: please check the 'trusted-keys' for '.' in named.conf.

Interesting!  As in, I've never seen that before.  The important part here
was the first few characters (the date): the CMOS reset had also reset the
system clock.  The time being off by 5 years was causing DNSSEC validation
to fail.

I messed around with 'date -s' trying to set the system clock, but gave up
figuring out how to specify the format correctly (I have used it in the
past, but always spend more time than I should staring at that man page;
it's always been a pet peeve of mine that 'date' outputs a format that 'date
-s' won't accept as input without a lot of extra work).  It should be easy
to just let NTP solve this right?  Well, NTP didn't work b/c all my servers
were names, not raw IP addresses, and since DNS failed...

Adding 4.2.2.1 as a temporary entry in resolv.conf allowed NTP to set the
clock correctly, and then everything was just smurfy.

So the good news is DNSSEC does seem to be validating things, and doesn't
"fail open" when it gets confused.  The other interesting thing I hadn't
thought about too much (b/c I never really looked too much into the details
of DNSSEC) is that it doesn't protect the last hop: my caching-dns server
needed DNSSEC to be correct, but if I acted as a pure client (by putting
another entry in my resolv.conf and bypassing my local dns-server) then no
DNSSEC needed.

That makes me a little more happy that I run my own DNS server: I can know
that DNSSEC validation is good at least until it gets to my server (as
opposed to the alternative, where it's only good until my ISPs or the
4.2.2.x/8.8.8.x set).

Matt