backups

[jack-rp9/bkPP+cDYtjvyW6yDsg at public.gmane.org (Jack Coats)]

After having spent half my career dealing with backups.  The wisest thing I
believe you can do is test it.

I have done this, and it seems to work for me:

Find an machine (almost anything for testing, if it is going to be a backup
or DR server, it should be roughly equivalent
to your production machine) with enough disk space, and restore your system
to it.

Document EXACTLY what you do, where you get the information/disks/CDs, etc.
and how you install them, and
how you get the data restored.

Once it is done, working, and tested, now fix your documentation.  And
personally I do it again using my new 'cookbook'
myself.  Iterate as needed to get it right.

Now, get a co-worker, friend, or consultant that has a good skill set,
understands tech talk, and buy them pizza and
Cokes while they go through your cookbook. ... Stay with them, document
everything they ask you, any issues they have,
etc.

Fix your documentation again, now, iterate the last step with a different
person.

Once you have no new 'fix' notes at the end of a test, you have a reasonable
cookbook.

Now repeat the test every six months, once with just you (or another local
admin) and one with a 'guest' (basically, not
you or someone that deals with these systems on a daily basis).

This verifies your procedures, that your backups can be restored, and your
customers can be served if you can't be there.

I have had instances where backups were bad, or the data needed was not in
the backups for whatever reason.
So I have eaten a lot of crow, and it doesn't go down well.  Performing the
tasks above helps ensure that you are
really ready if something bad happens.  The worst I have had to do is DR
tests (disaster recovery tests), and I have even
had to rebuild my production backup server from backups (very nerve racking
at the least).  Rebuilding mail servers is
bad, but it works (it also helps to make sure you have secondary email
servers in a different geography, like half way
across the country - they accept email for your domain, but users still
can't get to it until your email server comes back
up and processes the dumps from the secondary servers, but at least mail
doesn't bounce.).

I hope this helps a bit.