[Discuss] lvm snapshot cloning

[bogstad at pobox.com (Bill Bogstad)]

On Wed, Oct 26, 2011 at 11:49 PM, Tom Metro <tmetro-blu at vl.com> wrote:
> markw at mohawksoft.com wrote:
>> Suppose you have a 1TB hard disk. How on earth do you back that up?
>> Think now if you want to pipe that up to the cloud for off-site backup.
>>
>> The snapshot device "knows"
>> what is different. You don't have to really backup 1TB every time, you
>> only have to backup the changes to it since the last full backup.
>
> If you are at the point of considering developing software to do this,
> instead of just using off-the-shelf solutions, then you should consider
> using inotify[1]. I believe using this library you could log to a
> database the inodes that have been altered over a given period of time,
> which another tool could then use to package up the data and send it to
> your local or remote backup server.

I'm not sure how inotify would help that much.   Mark seems to be
focused on large files (database files, VM images).  He knows they've
changed, but for efficiency reasons he wants to replicate/backup only
the blocks that have changed due to the files' large sizes.  His
proposed solution is to piggyback on LVM snapshots which automatically
keeps track of changes at a block level granularity.  As I see it,
he's trying to bring back the functionality of the old dump/restore
backup programs except in a filesystem agnostic way.   Maintaining
dump/restore became intractable when the number of filesystems
exploded.  Tracking LVM is likely to be easier.

Your suggestion of DRBD seems like another way to go.   Although as
you note, the real-time nature of the data stream is an issue.  Using
LVM to consolidate changes to a single block is a possible advantage.

Of course, this is all based on guesswork by me about Mark's goals...

Bill Bogstad