[Discuss] file system checksums

[tmetro+blu at gmail.com (Tom Metro)]

Edward Ned Harvey wrote:
>Tom Metro wrote:
>> Why not use an error correcting file system on them?
> 
> It's difficult for me to build a Mac OSX Installation USB fob (or ubuntu
> install fob) that uses ZFS in the backend.   ;-)

Sure, makes sense.


> What other filesystems do checksumming anyway?  Not Mac, not NTFS, not
> ext3/4...

JFFS2 (CRC) http://sourceware.org/jffs2/jffs2-html/node3.html (an FS
made specifically for Flash, http://en.wikipedia.org/wiki/Jffs2)

Ext4 supposedly checksums its writes to the journal
(http://en.wikipedia.org/wiki/Ext3#cite_note-34), which is a step closer.

Ext2c is a hacked ext2 that adds checksums.
http://pages.cs.wisc.edu/~remzi/Classes/736/Spring2005/Projects/Brian-Chloe/ext2c.pdf

This article specifically addresses some of the reasons why Linux file
systems have problems with corruption on Flash:

Ext3 and RAID: silent data killers?
http://lwn.net/Articles/349970/

You could always split the Flash drive into two equal partitions and use
them as a RAID1 set. :-)  Not only will you get the checksums you want,
but also redundancy to repair the problem.

You just need to port mdadm to Windows and OS X. :-)


> ...corrupted bits from USB fob were from OS installation...ubuntu.
> The thing is ... Generally speaking, you don't know that corruption
> has occurred.

My recollection is that the Ubuntu install image includes a menu option
to perform an integrity check on the installation files.

Of course if your Flash drive is really flaky, even if the integrity
test passes, you may still read back corrupt bits when it comes time to
read the data for real.

 -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/