[Discuss] More Fun in ZFSland

[richard.pieri at gmail.com (Richard Pieri)]

On 5/16/2012 10:11 PM, Richard Pieri wrote:
> I'm not really paying for the encryption.  My N40L is I/O bound at
> the network interface so I can't saturate the disks or CPU over SMB
> and AFP.

It turns out that I was very much in error on this.

My level 0 backup last weekend took about as long as I expected since 
it's I/O bound by USB2.  But then the weekly scrub took several times 
longer than the unencrypted scrub and that seemed very odd to me. 
Nothing seemed obviously wrong.  System load was fine but the scrub 
throughput was very low.  So I installed the ZFS on Linux kernel module 
set to take FUSE out of the loop and did some testing.  And then some 
Google searching proved what I found.  kcryptd is stupid.

dm-crypt sets up a kcryptd thread for each encrypted device, but kcryptd 
itself isn't SMP-aware.  All of those threads run together on one core. 
  When it peaks at 100% usage then that's the I/O throughput cap. 
There's simply no way to make it go any faster.

Time to revert and rethink.  Live and learn and pass on that knowledge. 
  I'm reverting using the offline/resilver method: zpool offline the 
dm-crypt device, remove it from /dev/mapper, make a symlink to trick ZFS 
into thinking the raw partition is the /dev/mapper device, zpool replace 
and let it resilver.  I want to see how reducing the number of dm-crypt 
devices affects throughput.  I also want to see resilver in action.

-- 
Rich P.