[Discuss] Backing up LVM partitions using snapshots

[markw at mohawksoft.com (markw at mohawksoft.com)]

>> From: discuss-bounces+blu=nedharvey.com at blu.org [mailto:discuss-
>> bounces+blu=nedharvey.com at blu.org] On Behalf Of Bill Bogstad
>>
>> On Wed, Dec 14, 2011 at 10:53 AM, Richard Pieri
>> <richard.pieri at gmail.com>
>> wrote:
>> > On Dec 13, 2011, at 11:11 PM, markw at mohawksoft.com wrote:
>> >>
>> >> Using a block level store, an incremental backup is no different than
>> a
> full
>> backup.
>> >
>> > You're not following me. ?You have a full dump. ?You have a first
> differential
>> against that dump. ?You have a second differential made against the
>> first
>> differential. ?If you lose the first differential then you are screwed.
>>
>> I've been watching the (second?) incarnation of this thread for a
>> while now and I think that I see your point.
>
> That's not how incremental snapshots work.  Well sure, if you leave these
> datastreams laying around as files, then the argument is true, lose the
> first incremental and you lose everything after it.  But that's not what
> you
> do.
>
> I can't speak on behalf of Mark's stuff, but there is only one sane way to
> behave:
>


Picture this scenario......

You have a number of LVM 500G volumes containing virtual machines cloned
from single ancestor. Free space is zero initialized.

You have a backup volume which is RAID and large.

First step, full backup:
1) Snapshot each volume.
2) Do a block level backup that does duplicate removal and compression on
each snapshot.

At the end, your backup will probably be much smaller than 500G because of
the duplicate removal.

You will be left, on your backup volume, a single data file that contains
the block data and a file block list for each volume.

Second and subsequent steps, incremental backup:
1) Create a new snapshot of each volume.
2) Take a change list from the previous snapshot and optionally delete it.
3) backup the changed blocks to the data file.

On the backup volume, your data file will have grown a bit, but you will
get additional full block list files for each volume.

You could even push out the block lists and incremental data to the cloud
for off-site backup.

So, even with a differential backup, you can get a full representation of
the whole or multiple wholes.

The problem I have with tape is how poor its reliability is. When I worked
at Sytron, a tape backup software company, the reality was (and still is)
that tape is not 100% reliable, hell, not even 99% reliable. Tape backups
are designed as a solution that accepts that if you have enough coverage,
you'll probably be safe. Tape always has a risk that data will be lost and
if you have enough tapes, your data is surely safe on one of them.

That model is changing. The EMCs, NetAPPs and the like don't rely on tape.
They rely on the sort of strategy I am describing. Duplicate reduction and
replication, not "backup."