[Discuss] ZFS for an Encrypted External Disk: My Experience and Notes

[dsr at randomstring.org (Dan Ritter)]

markw at mohawksoft.com wrote: 
> Gotta ask again. Why mirror and not "RAID?"

The tradeoffs are:

	read IOPS.
	read xfer
	write IOPS
	write xfer
	capacity loss
	recovery after 1 disk lost
	recovery after 2 disks lost

If you have four disks, your choices are: stripe, mirror, stripe
over mirrors, RAID5 (3+1p) and RAID6 (2+2p).

stripe: read IOPS=1xbase disk. read xfer=4x. write IOPS=1x.
        write xfer= 4x. No capacity loss. No recovery from 1 disk loss.

mirror: read IOPS=4x. read xfer=4x. write IOPS=1x. 
        write xfer=1x. Capacity loss is 75%. Recovers from 1 loss by reading
        and writing 1 disk's worth of data. Recovers from 2 and 3 disk
        losses the same way.

stripe over mirrors: read IOPS=2x. read xfer=2x. write IOPS=1x.
        write xfer=2x. Capacity loss is 50%. Recovers from 1
        loss by reading and writing 1 disk's worth of data.
        Recovers from 2 losses only if they are in different
        mirrors.

RAID5: read IOPS=1x. read xfer=3x. write IOPS=up to 1x
       (parity must be calculated). write xfer=3x. Capacity loss
       is 25%. Recovers from 1 loss by reading 3 disks of data
       and writing 1 disk of data. No recovery from 2 disks.

RAID6: read IOPS=1x. read xfer=2x. write IOPS=up to 1x. write
       xfer=2x. Capacity loss is 50%. Recovers from 1 loss by
       reading 2 disks and writing 1 disk. Recovers from 2 losses
       by reading 2 disks and writing 2 disks.


So, what's most important to you?

Let's rule out striping by itself. If you want pure speed, buy
NVMe PCIe4 storage. You get to pay for it, but it's awesome.

If you desperately care about survivability, I don't think a
4-way mirror is as sensible as buying two complete systems and
having a 2-way mirror in each.

RAID5 has a really slow recovery, and can't (in this 4 disk
scenario) recover from a 2-disk loss. If you are building a pure
backup server, maybe this is a good move for you.

So that leaves stripe over mirror and RAID6. They both cost you
50% of capacity. SoM gives you better read IOPS, which in my
experience is what humans notice first in interactive usage.
Both can recover from 2 disks being lost, but RAID6 takes longer
to recover. SoM only recovers from 2 disks being lost if they
are in different mirrors.

Stripe over mirror gives me the best tradeoffs for my situation:
primary data storage for a bunch of applications, plus backup
duty for some other machines, optimizing for recovery time when
1 disk crunches.

-dsr-