[Discuss] lvm snapshot cloning

[matt at mattshields.org (Matt Shields)]

On Tue, Oct 25, 2011 at 10:16 PM, Richard Pieri <richard.pieri at gmail.com>wrote:

> On Oct 25, 2011, at 7:51 PM, markw at mohawksoft.com wrote:
> >
> > The snapshot has no effect on the master, and yes, we've already said and
> > we already know it is a weakness in LVM that if you don't extend your
> > snapshots you lose them. This can be mitigated by monitoring and
> automatic
> > volume extension.
>
> You missed it.  This isn't about what happens to master.  It's what happens
> to b when a disappears.  If master<-a<-b and a disappears due to reaping
> then b becomes useless.  Or b is reaped, too.  Either way you're dealing
> with data loss.  This is why LVM will not do what you originally asked
> about.
>
> Monitoring has problems.  If the volume fills up faster than the monitor
> polls capacity then you lose your data.  If the volume fills up faster than
> it can be extended then you lose your data.  If the volume cannot be
> extended because the volume group has no more extents available then you
> lose your data.  Like I wrote at the start: LVM will quite happily bite your
> face off.
>
> Now, to address your most recent question:
>
> How do I back up a 1TB disk.  Think about this: how do you intend to do a
> restore from this backup?  The most important part of a backup system is
> being able to restore from backup in a timely fashion.
>
> I have in production a compute server with two 8TB file systems and a 9TB
> file system, all sitting on LVM volumes.  I have an automated backup that
> runs every night on this server.  It's an incremental file system backup so
> I'm only backing up the changes every night.  This is, as you might expect,
> quite faster than trying to do full backups of 25TB every night -- which I
> can't because it would take three days to do it.
>
> On smaller capacity volumes, in the several hundred GB range, I use
> rsnapshot to do incremental file snapshots to a storage server.  Again, I
> don't back up the raw disk partitions every time.  I only back up the
> changed files.
>
> In both cases -- and in fact with all my backups -- they are file level
> backups.  The reason being that if I need to restore a single file or
> directory then I don't have to rebuild the entire volume to do so.  I can
> restore as little or as much as I need to recover from a mistake or a
> disaster.
>
> Suppose the case of a live volume that needs to be in a frozen state for
> doing a backup.  Database servers are prime examples of this.  Here, I would
> freeze the database, make a snapshot of the underlying volume, and then thaw
> the database.  Now I can do my backup of the read-only snapshot volume
> without interfering with the running system.  I would delete the snapshot
> when the backup is complete.
>
> If I were using plain LVM and ext3 for my users' home directories then I
> would do something similar with read-only snapshots.  There would be no
> freeze step, and I would keep several days worth of snapshots on the file
> server to make error recovery faster than going to tape or network storage.
>  As it is, I use OpenAFS which has file system snapshots so I don't need to
> do any of this and users can go back in time just by looking in .clone in
> their home directories.  I still have nightly backups to tape for long-term
> archives.
>
> Now, time to poke holes in your proposal.  I have a physics graduate
> student doing his thesis research project on a shared compute server along
> with a dozen others.  They collectively have 7.5TB of data on there.  This
> is a real-world case on the aforementioned compute server.  Said student
> accidentally wipes out his entire thesis project, 200GB worth of files.
>  It's 9:30 PM and he needs his files by 8am or he fails his thesis defense,
> doesn't graduate and I'm looking for a new job.
>
> With my file level backup system I can have his files restored within a
> couple of hours at the outside without affecting anyone else's work.
>
> With your volume level backup system I would spend the night on Monster
> looking for a new job.  The problem with it is that I can't restore
> individual files because it isn't individual files that are backed up.  It's
> the disk blocks.  I can't just drop those backed-up blocks onto the volume.
>  Here:
>
>  master->changes->changes->changes
>       \->backup
>
> If I dumped the backup blocks onto the volume then I'd scramble the file
> system.  Restoration would require me to replicate the entire volume at the
> block level as it was when the backup was made.  This would destroy all the
> other researchers' work done in the past however many hours since that
> backup was made.  I would fire myself for gross incompetence if I were
> relying on this kind of backup system.  It's that bad.
>
> It gets worse.  What happens when the whole thing fails outright?  Total
> disaster on your 1TB disk.  Now it's not just 29 minutes to restore last
> night's blocks.  It's two hours to restore the initial replica and then 30
> minutes times however many deltas have been made.  Six deltas means 5 hours
> to do a full rebuild.  I can do a complete restore from TSM or rsnapshot in
> half that time, maybe less depending on how much data was actually there.
>
> Data point: It takes ~19 hours to restore 7.5TB from enterprise-class
> Tivoli Storage Manager over 1GB Ethernet to a 12x1TB SATA (3Gb/s) RAID 6
> volume.  I had to do it this past spring after the RAID controller on that
> volume went stupid and corrupted the whole thing.
>
> --Rich P.
>
> _______________________________________________
> Discuss mailing list
> Discuss at blu.org
> http://lists.blu.org/mailman/listinfo/discuss
>

Previously I had mentioned how I used LVM for snapshotting large live MySQL
databases and also creating QA snapshots from live DB replication.  Yes,
there are issues with the snapshot space filling up, but that's usually
because people haven't calculated how much space to leave available which it
totally dependent on you know how long you'll need the snapshot around.  If
I have a 1TB volume and configure 950GB for data and 50GB free for LVM, but
I have 100GB of changes daily, then I'm going to have issues with running
out of space.  You should know how your blocks are used before blindly using
LVM for snapshots.  If in doubt always reserve extra space for the
snapshots.

Second, if you are going to use LVM, I suggest setting up monitoring in some
way.  Nagios has a plugin
http://exchange.nagios.org/directory/Plugins/Uncategorized/Operating-Systems/Linux/check_lvm/details

All that said, I've had extremely good luck in using LVM.  I haven't noticed
any performance issues, although I should point out that I use RAID1 or
RAID10 sets of 15k SAS drives.

I've also used SAN's by Dell/EMC and Compellent(now Dell).  I had a
Compellent engineer explain the way that most SAN's do snapshotting is very
similar to the way LVM works where it leaves your original data alone when
snapshotted, then all changes to the original data and the snapshot go into
the unused/unallocated space.  Of course SAN's are more optimized for doing
this.

Matthew Shields
Owner
BeanTown Host - Web Hosting, Domain Names, Dedicated Servers, Colocation,
Managed Services
www.beantownhost.com
www.sysadminvalley.com
www.jeeprally.com
Like us on Facebook <http://www.facebook.com/beantownhost>
Follow us on Twitter <https://twitter.com/#!/beantownhost>