Source
r/zfs•
CatProgrammer

Characterizing files for metadata/etc. before transfer to ZFS pool

https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954 contains info for getting metadata size and block-size histograms, but both processes assume the data is already in a ZFS pool. I've looked around but can't seem to find any techniques for getting that info before actually moving those files to ZFS (useful when first setting up a ZFS system). The histogram can be approximated by a simple analysis of file sizes that rounds each up to a power of two (with whatever cutoff is wanted for record size), but I'm not sure how to emulate the metadata calculations. That link does mention that the metadata will be about 0.3% of a "typical" workload, but it does not clarify what a typical workload is, and I assume that will also vary based on chosen record size, how often files are modified (due to COW), and the like.

Posted Jun 20, 2022, 6:05 PM UTC · edited

39
Dagger0•

I did some measurements on this. Metadata is something like the sum of:

a) 1 GB per 100k multi-record files.
b) 1 GB per 1M single-record files.
c) 1 GB per 1 TB (recordsize=128k) or 10 TB (recordsize=1M) of data.
d) 5 GB of DDT tables per 60 GB (recordsize=8k), 1 TB (recordsize=128k) or 10 TB (recordsize=1M) of data, if dedup is enabled.
e) plus any blocks from special_small_blocks.

redundant_metadata=most should halve (a) and (c).

Take these with a grain of salt, do your own measurements etc, but I've posted these a few times before and not had anybody point out any major problems, so I think they should be reasonable as a rough estimate.

3
CatProgrammerOP•

Those are definitely some more clear approximations, thanks.

1
WantsTheZFS•

I believe the command you want to run is zdb, I don’t recall all the options, but I’ve used it before to profile a dataset. If the data isn’t in a zfs dataset then you might be out of luck for zfs tools helping you profile anything.

1
CatProgrammerOP•

That's the one used in the linked post, yes. I am specifically asking about alternatives for profiling files on a non-Z fileystem before moving them to ZFS to make setup of the ZFS-based system easier.

1
mercenary_sysadmin•

It might help if you could be clear about what "setup of the ZFS-based system" you want to do based on whatever it is you find out about your current data.

3
CatProgrammerOP•

Just looking into sizing a special vdev (mirrored SSDs) to go with a spinning rust array that could have some directories with lots of children that could be modified/added to now and then (so fast metadata storage seems useful), and in some places a mix of very large and very small files, where the small files may be modified as well (so small blocks storage seems useful). I'd be fine with just not putting in a special vdev at first and seeing how the performance goes without it as it's not intended for data center use, so I'm not sure how much improvement I'd actually get. (I do have support for >gigabit networking but it's not going to have lots of simultaneous users just yet.) However, everything I've read states that only new stuff gets put on a special vdev. Thus, to put one in later I'd have to make a backup and then restore from that backup using rsync/some other non-ZFS copy in order to actually get the benefit for data already present, so it'd be nice to figure out if it would provide some benefit ahead of time.

1
mercenary_sysadmin•

You would not need a non-zfs copy operation to get data and metadata onto the special; replication works just fine for that.

I generally advise against the special. I've seen some people say it helped them a lot, but in my own testing I've had a lot of trouble seeing it make much real difference. I suspect it's most helpful with pools that have either been allowed to become and remain EXTREMELY full for long periods, or pools with enormous numbers of very small files.

For a fairly general purpose small scale fileserver, I don't see much utility.

1
ElvishJerricco•

rounds each up to a power of two (with whatever cutoff is wanted for record size)

FYI if you have any kind of ZFS compression enabled (even just zle), it doesn't have to be a power of two. Just a multiple of 2^ashift

1