[Discuss] On Btrfs raid and odd-count disks

[richard.pieri at gmail.com (Richard Pieri)]

Today I ran into a problem that I hadn't expected but I should have
expected.

Remember that Btrfs "raid" replicates file data and metadata, not disk
blocks. If you have three disks in a raid1 configuration then any file
written to one disk will have a replica written to another disk. If you
have 3 times 500G disks then you have ~700G usable capacity. df reports
this as 1.2T since it doesn't fully understand Btrfs.

Mostly.

Say you have 500G disks in a 3-disk raid set, and you've stored 150G of
data. df will show 300G used and 1.1T free. That's 550G usable after
dividing by half for mirroring. The largest file that you can write is
still only 400G. This assumes even balancing of that 300G across all of
the disks in the set. If that 300G is a single 150G file which is
replicated across two disks in the set then the largest file that can be
written is 350G -- the space available on those two disks.

And if you do fill up one of the disks, such as by using dd like I did,
then you will start getting file system full errors despite df showing
plenty of usable space.

-- 
Rich P.