admins worst nightmare...

[bogstad-e+AXbWqSrlAAvxtiuMwx3w at public.gmane.org (Bill Bogstad)]

On Mon, Mar 8, 2010 at 3:43 PM, Stephen Adler <adler-wRvlPVLobi1/31tCrMuHxg at public.gmane.org> wrote:
> Guys,
>
> I discovered that my 6 terabyte file does not seem to be working
> properly... I did a copy of a gigabyte sized file to find that the
> original and copied md5sum's to differ.... uggg.... I'm doing a
> filesystem check right now, but I'm wondering if you guys have any
> thoughts on what may be going on with the file system. It's an ext3 file
> system mapped over a software raid 5 raid array. When I created the file
> system, I used all the default mkfs parameters for a journaled file
> system. (i.e. mkfs -J /dev/md127; where md127 is my raid device.)
>
> When I checked a small file, several megabytes in size, the copy and
> original had the same md5sum.

A few suggestions:

Check your log files to see if the system is complaining about anything.

Look into smartctl to see what the drives themselves think of their health.

The problem may not be disk or filesystem related.  Depending on where
you are doing your md5sum, it could be RAM in the fileserver or
client, network corruption, etc.

Try md5suming the original file multiple times and see if it changes.
The question here is what part (of a complicated system) is actually
introducing the errors.  Try it from a different client system. etc.
...

Good Luck,
Bill Bogstad