Source
r/zfs•
killamongaro

Scrubbing method, really good performance, confused as to why

So I ran my first scrub job the other day and was seeing my disks hitting close to 2000 IOPS. These are consumer grade, WD Blue's, 7200 RPM disks backing the pool, so 2000 IOPS is like 5 times what I would expect to see from the disks. I figured a possible cause would be that scrub jobs are done using sequential reads, and since apparently my pool was 100% healthy there was no need to write. Been looking around for an explanation but not finding anything really.

Usually I see people complaining about the performance of scrubs, but opposite for me I guess.

Posted Jul 24, 2015, 1:32 AM UTC

515
Virtualization_Freak•

Scrubs are chronological as they walk through the block tree.

64KB blocks at 2k IOPS = 128MB/s.

If you haven't changed your data much (or at all) and it's all large files, it's doable.

6
killamongaroOP•

That's interesting, thank you for the information. It is indeed all large files that get written once and read (Plex server) so very little change, except when I do a binge download. I used to have a virtual machine storage pool, but I nuked that a long time ago due to stability issues, so there isn't really any high traffic data sitting on my array.

1
Virtualization_Freak•

Although thinking about it, you didn't mention which OS.

FreeNAS max block size is 128KB. What's your actual scrub speed in MB/s?

Written once and never changed is what allows for high speed scrubbing.

1
killamongaroOP•

Oh, right I'm using FreeBSD, and if I'm calculating right it looks like around 888, but I'm pretty sure that's Mbps, so more like around 100-125 MB/s. I guess technicallythe SATA connections can handle 3GB/s, but the IOPS there would mean all the data would have to be read from cache.

1
MadPhoenix•

You should see the scrub throughput when running 'zpool status <pool>'. What does your pool layout look like (# of disks and vdev types)? It's hard to judge what scrub throughput you should be seeing without knowing the pool makeup.

Also, note that SATAII is 3 Gigabit, not Gigabyte, per port. This shouldn't be a bottleneck with spinning disks unless you have expander backplanes where one SATA port is serving multiple disks.

1
txgsync•

It's really common to vastly exceed IOPS expectations using ZFS. As a trivial example, imagine you sequentially-wrote a bunch of 8kb sectors to disk, and perform a sequential read using a new generation HGST 8He drive with 205Mbyte/sec throughput. It will read these mostly-linear at around 26,240 IOPS, more or less.

Conversely, using SLOG devices for writes at work, I'll commonly see "random" write IOPS pushing 300+ on 7200RPM drives for even large blocks. That's because a SLOG device coalesces writes into a linear write; 300 IOPS 128k is only 38mbyte/sec: well within the throughput limits of a spinning platter at those RPMs.

The worst case for scrubs is when you have a lot of very small data (4k, 8k, or smaller) with a RAIDz2 or -- God forbid -- RAIDz3 pool that has exceeded about 96% full at some point in its life, and you haven't used a SLOG device. Then your intent log will have checkerboarded the pool, and subsequently filled it in, and you'll have a really patchy-looking bunch of metaslabs that take forever to iterate through sequentially by individual TXGs...

Disclaimer: My opinions do not necessarily reflect those of Oracle or its affiliates.

1
killamongaroOP•

Interesting. I had to do a bit of research to understand some of this, but it makes sense and explains a lot of what's going on and how ZFS works when it writes data. I don't have a SLOG device, but I don't write often, and it's not a big deal if it takes a while to do writes because I'm not really looking for high performance read/write, just enough to stream, and my crap CPU is by far the bottle neck there.

1
MadPhoenix•

That's because a SLOG device coalesces writes into a linear write

That's not how it works. The SLOG is only an intent log, it's not in the data path between RAM and disk like a RAID write cache is. The performance gain for a SLOG is just that sync writes return much faster because you're journaling to a fast device rather than disk.

Then your intent log will have checkerboarded the pool, and subsequently filled it in, and you'll have a really patchy-looking bunch of metaslabs that take forever to iterate through sequentially by individual TXGs...

Wat? It's not like you're writing from the SLOG to the pool ... you write from RAM to the pool. Source

... you work for Oracle?

1
txgsync•

Nothing you just said is incorrect. Nothing I said is incorrect either. The txg_sync is from RAM. A SLOG device allows those writes to coalesce somewhere other than the pool's spindles and blocks, instead to a dedicated VDEV. The SLOG never needs to be read unless you suffer certain types of failures.

Intent logs checkerboard pools in absence of SLOG and in the presence of fairly full pools. The context for your second response was that if you do NOT use a SLOG device, you may suffer intent log checkerboarding on your disk...

I think perhaps you've misunderstood my pithy example. My intent was not to explain so much how the intent log works, but the impact of certain design choices under some conditions.

1
MadPhoenix•

Any pool is going to have performance problems at 96% full. The recommendation is to only allocate 80-90% of pool capacity.

I guess my issue is you're using the term "coalescing" which has a very specific meaning in the context of storage subsystems - the reordering and combining of IOs between RAM and disk to achieve better throughput (sometimes at the cost of higher latency). There's nothing different about how ZIL writes are ordered or merged when you write it to an SLOG rather than a VDEV in the pool.

Conversely, using SLOG devices for writes at work, I'll commonly see "random" write IOPS pushing 300+ on 7200RPM drives for even large blocks. That's because a SLOG device coalesces writes into a linear write; 300 IOPS 128k is only 38mbyte/sec: well within the throughput limits of a spinning platter at those RPMs.

Here, you seem to be implying that an SLOG will make a 7200 RPM disk capable of 300+ IOPS. That's not the case. If ZIL writes go to a disk VDEV, they're cheap, large, sequential writes. The only case where you'll see higher pool IOPS due to an SLOG is with a largely/entirely synchronous workload, since in that case the writing process will block until ZFS gets the transaction onto the ZIL. If you take the same pool and ran a purely asynchronous workload against it, there'd be no measurable difference between a pool with an SLOG and one that writes it's ZIL to a vdev.

Here's a better explanation from source:

Quick recap: the ZIL or ZFS intent Log is - as I understand it - only relevant for synchronous writes. If data integrity is important to an application, like a database server or a virtual machine, writes are performed synchronous. The application wants to make sure that the data is actually stored on the physical storage media and it waits for a confirmation from ZFS that it has done so. Only then will it continue. Asynchronous writes on the contrary, never hit the ZIL. They are just cached in RAM and directly written to the VDEV in one sequential swoop when the next transaction group commit will be performed (currently by default every 5 seconds). In the mean time, the application gets a confirmation of ZFS that the data is stored (a white lie) and just continues where it left off. ZFS just caches the write in memory and actually write the data to the storage VDEV when it feels like it (fifo). As you may understand, asynchronous writes are way faster because they can be cached and ZFS can reorder the I/O to make it more sequential and prevent random I/O from hitting the VDEV.

1
txgsync•

A 7200RPM disk is absolutely, 100% capable of 300+ IOPS if they are linear. I observe this every single day. Just not 300 RANDOM write IOPS, which a SLOG device helps one avoid because the TXG is cached in RAM for up to 5 seconds (by default; can be longer).

A dedicated SLOG device is what allows this to happen: if the intent log were competing for writes on the spindle, one couldn't observe such a direct trade-off by leveraging the SSD as stable storage until the TXG could commit.

The doc you cited is wrong about asynchronous writes never hitting the intent log. Certain ZFS structures will continue to use the intent log, though the data blocks themselves will not. I am, however, typing this on a phone at a remote camp in the wilderness and lack the patience to elaborate :)

1