[Discuss] Debian 11 -> 12

[richard.pieri at gmail.com (Rich Pieri)]

On Wed, 22 May 2024 21:22:30 -0400
Steve Litt <slitt at troubleshooters.com> wrote:

> True. I never thought of lots of NVMe drives.

It's not a common use case. One tends not to find compute servers
loaded with PCIe cards each worth at least as much as the entire rest
of the server.

>>I have seen filesystems (notably XFS and early ext4) lose or damage  
> I've never seen ext4 do anything worse than recover from journal.

It was a high performance compute/storage server for analyzing LHC
data. Had a big for the time storage array, 18TB or 24TB, mdadm for
RAID, ext4 filesystem. Within a week it had thoroughly corrupted it's
filesystem metadata. I restored a backup, verified it was good... and
it did the same thing within a few days. Remade the filesystem as
either ext3 or XFS, I don't remember which but probably XFS, was
running fine for three+ years when I left that gig. Only change was
replacing ext4 with something else so I'm confident it wasn't the
hardware or the OS. This was around... 2012 and ext4 was still
relatively new and not yet extensively stress-tested.

While at my current employer, we found an obscure DirectIO data
corruption bug in ext4. Only happened under a very specific and niche
condition. Okay, technically it was one of our customers who discovered
it. We identified exactly where it was.

The XFS bug was similarly obscure with a similarly specific and niche
condition. As was the VFS bug. We, and our customers, do a lot of odd,
weird, niche things.

> My concern is recovering from user error. The fewer layers of
> abstraction, the easier it is.

Maybe. Depends on the mistake. Running newfs instead of growfs, a
mistake I did once make, didn't care about the volume manager and would
have required the same fix regardless: restore from backup.

Though I did make lemonade out of that mistake. It gave me the
opportunity to reconfigure the underlying storage array for better
performance and a little more capacity: one large RAID6 volume instead
of two smaller, differently-sized RAID6 volumes.


> By the way, this isn't discussed in this thread, but my finding is
> that formatting thumb drives ext4 instead of leaving them at that
> windows format makes them *much* more reliable.

ext4/3/2 are terrible filesystems for FTL (flash translation layer)
storage. Use a filesystem designed for FTL media: exFAT for
portability, f2fs for linux-only use. Better still, don't expect
reliability from USB flash or SD cards at all. Treat them like
diskettes but bigger and faster and a bit more durable, but just as
disposable.

-- 
\m/ (--) \m/