[Discuss] lvm snapshot cloning

[richard.pieri at gmail.com (Richard Pieri)]

On Oct 25, 2011, at 7:51 PM, markw at mohawksoft.com wrote:
> 
> The snapshot has no effect on the master, and yes, we've already said and
> we already know it is a weakness in LVM that if you don't extend your
> snapshots you lose them. This can be mitigated by monitoring and automatic
> volume extension.

You missed it.  This isn't about what happens to master.  It's what happens to b when a disappears.  If master<-a<-b and a disappears due to reaping then b becomes useless.  Or b is reaped, too.  Either way you're dealing with data loss.  This is why LVM will not do what you originally asked about.

Monitoring has problems.  If the volume fills up faster than the monitor polls capacity then you lose your data.  If the volume fills up faster than it can be extended then you lose your data.  If the volume cannot be extended because the volume group has no more extents available then you lose your data.  Like I wrote at the start: LVM will quite happily bite your face off.

Now, to address your most recent question:

How do I back up a 1TB disk.  Think about this: how do you intend to do a restore from this backup?  The most important part of a backup system is being able to restore from backup in a timely fashion.

I have in production a compute server with two 8TB file systems and a 9TB file system, all sitting on LVM volumes.  I have an automated backup that runs every night on this server.  It's an incremental file system backup so I'm only backing up the changes every night.  This is, as you might expect, quite faster than trying to do full backups of 25TB every night -- which I can't because it would take three days to do it.

On smaller capacity volumes, in the several hundred GB range, I use rsnapshot to do incremental file snapshots to a storage server.  Again, I don't back up the raw disk partitions every time.  I only back up the changed files.

In both cases -- and in fact with all my backups -- they are file level backups.  The reason being that if I need to restore a single file or directory then I don't have to rebuild the entire volume to do so.  I can restore as little or as much as I need to recover from a mistake or a disaster.

Suppose the case of a live volume that needs to be in a frozen state for doing a backup.  Database servers are prime examples of this.  Here, I would freeze the database, make a snapshot of the underlying volume, and then thaw the database.  Now I can do my backup of the read-only snapshot volume without interfering with the running system.  I would delete the snapshot when the backup is complete.

If I were using plain LVM and ext3 for my users' home directories then I would do something similar with read-only snapshots.  There would be no freeze step, and I would keep several days worth of snapshots on the file server to make error recovery faster than going to tape or network storage.  As it is, I use OpenAFS which has file system snapshots so I don't need to do any of this and users can go back in time just by looking in .clone in their home directories.  I still have nightly backups to tape for long-term archives.

Now, time to poke holes in your proposal.  I have a physics graduate student doing his thesis research project on a shared compute server along with a dozen others.  They collectively have 7.5TB of data on there.  This is a real-world case on the aforementioned compute server.  Said student accidentally wipes out his entire thesis project, 200GB worth of files.  It's 9:30 PM and he needs his files by 8am or he fails his thesis defense, doesn't graduate and I'm looking for a new job.

With my file level backup system I can have his files restored within a couple of hours at the outside without affecting anyone else's work.

With your volume level backup system I would spend the night on Monster looking for a new job.  The problem with it is that I can't restore individual files because it isn't individual files that are backed up.  It's the disk blocks.  I can't just drop those backed-up blocks onto the volume.  Here:

  master->changes->changes->changes
       \->backup

If I dumped the backup blocks onto the volume then I'd scramble the file system.  Restoration would require me to replicate the entire volume at the block level as it was when the backup was made.  This would destroy all the other researchers' work done in the past however many hours since that backup was made.  I would fire myself for gross incompetence if I were relying on this kind of backup system.  It's that bad.

It gets worse.  What happens when the whole thing fails outright?  Total disaster on your 1TB disk.  Now it's not just 29 minutes to restore last night's blocks.  It's two hours to restore the initial replica and then 30 minutes times however many deltas have been made.  Six deltas means 5 hours to do a full rebuild.  I can do a complete restore from TSM or rsnapshot in half that time, maybe less depending on how much data was actually there.

Data point: It takes ~19 hours to restore 7.5TB from enterprise-class Tivoli Storage Manager over 1GB Ethernet to a 12x1TB SATA (3Gb/s) RAID 6 volume.  I had to do it this past spring after the RAID controller on that volume went stupid and corrupted the whole thing.

--Rich P.