[Discuss] ZFS scrub performance degrading

[markw at mohawksoft.com (markw at mohawksoft.com)]

> Configuration: 16GB RAM, 64GB NVMe L2ARC, 4x6TB RAIDZ2 zpool. Earlier
> this year I bumped the server from 16GB to 64GB RAM, and I removed the
> L2ARC. I made no other hardware changes.
>
> Prior to the upgrade the monthly scrub process took 15-20 hours and the
> four disks would run at their full performance of around 130MB/s
> for the duration. Now, read performance starts dropping at around the
> 50% completion mark down to under 40MB/s by the time it finishes.
>
> If I pause the scrub, reboot, and resume, then I get the full read
> performance again, and it will finish the scrub in a reasonable time.
> Adding the L2ARC back does not make any appreciable difference.
>
> Any suggestions about what to look at? I'm genuinely confused by this.

Yea, ZFS has a lot of ways to be slow. My first concern is that you should
check the drives individually to ensure that are not slowing down. I've
seen that just before losing drives.

Rebooting and seeing a restored speed could be scheduling issue where the
background scrub process is competing with other tasks. After a reboot
fewer processes may be competing. There are a couple settings for this.

Could there be a lot of I/O going on in addition to your scrub? Rebooting
may eliminate a lot of processes using I/O because they don't get
restarted.

What version of ZFS?

What is the bus technology to the disks? SATA, SAS?

How many PCI lanes do you have to the controllers? What else in on the
controller?

Is sync enabled?

Are you sharing anything from the ZFS pool over a network? iscsi and a
zvol?  NFS?

Is compression enabled? Which type? What's the compression ratio?

What does zpool iostat have to say when it is running fast vs running
slow? How does that equate to system iostat?