Boston Linux & Unix (BLU) Home | Calendar | Mail Lists | List Archives | Desktop SIG | Hardware Hacking SIG
Wiki | Flickr | PicasaWeb | Video | Maps & Directions | Installfests | Keysignings
Linux Cafe | Meeting Notes | Blog | Linux Links | Bling | About BLU

BLU Discuss list archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Discuss] Backing up LVM partitions using snapshots

On Dec 13, 2011, at 6:30 PM, Edward Ned Harvey wrote:
> In my situation, I have millions of files which are mostly static.  So
> simply walking the filesystem to find which files changed is the big time
> consumer.  We were formerly backing this up via rsync, and it ran 10-12
> hours per night.  Then we switched to ZFS and now the incremental takes
> typically 7 minutes.  Mark's LVM snapshots should be able to achieve similar
> results on this particular dataset.

Probably faster still, as there is no file system walking to be done.  See also the DRBD.  It and Mark's backup method have some similar ideas.

I see this as an example of a limitation of traditional UFS-derived file systems and why you may want to consider a modern, high-performance file system rather than ext2/3/4 with Mark's backup system.  The bottleneck is the stat() function.  It's slow on traditional UFS-style file systems.  We've known it since the heyday of Usenet.  Replacing a traditional directory structure with a database offers massive performance improvements as you've seen with your conversion to ZFS.  These improvements carry over to almost all file operations, not just backups.

--Rich P.

BLU is a member of BostonUserGroups
BLU is a member of BostonUserGroups
We also thank MIT for the use of their facilities.

Valid HTML 4.01! Valid CSS!

Boston Linux & Unix /