Source
r/zfs•
SS_Mike

Stuck on a ZFS deployment project plan - help?

I will warn you that I am not overly familiar with ZFS but I have some guys that work for me that will be setting this up that know all the details, I only know the concepts so I apologize in advance if I use incorrect terms. I just want to bounce this off some people with experience with this size ZFS deployment before I put this project plan together.

I'm getting ready to roll out a new server and 48TB (12x4TB) DAS for offsite backups for clients (about 50 active connections at night but delta and staggered) and minor internal storage (~3TB). The server will have dual 6-core Xeons, 64GB of ECC RAM, 2x300GB 10k mirror for the OS, an SSD for L2ARC, and an SSD for ZIL, plus the DAS. I expect to see about 600GB of moving data per month after initial setup.

I've been thinking about FreeBSD for ZFS to present to my small cluster via iSCSI for the VMs to pick up, but this server will also have the backup software installed on it to facilitate the offsite backups. Although this will be a secondary storage pool for the backups (software handles the back end sync) it is still internal and customer data so I am very concerned about corruption. I'm looking at ZFS to ease my concerns but I don't know about having that many drives in a vdev. I was thinking one big Z3 but I keep seeing references to the recommended max for a Z3 being 9 drives. So in the interest of maximizing available space but also tolerating failure - how would you set this up? (2) 6-disk Z2 vdevs in the zpool? Any size or type recommendations on the SSDs? I was thinking 250GB MLC of some sort. I can also add RAM if you think I should. Thank you!

Posted Jun 18, 2016, 5:05 AM UTC

419
parawolf•

For our database backups we use stripe and mirror everything on the backup side. No l2arc and no zil. Storage set to throughout logbias. We backup over multiple ten gig links from the ZFS side, with multiple database servers with a single ten gig link for backups.

Works a treat. We get a few hundred megabytes/sec throughput recorded during backups. Though we have 80 disks in our pool, 78 disks and 2 as hot spares.

Consider that this backup system needs to be available and since its backup, it won't be watched as hard as production. Make sure you have all your alarms set, tested and action plans in place if you are building it yourself.

1
SS_MikeOP•

Yeah I was hoping not to lose 50% of available storage but it might be necessary. This will be a production server for us and will get the same monitoring as the rest of our servers. How much memory do you have in that? Wish I had enough hardware for that many drives!

1
parawolf•

Ours are Oracle ZS4. So 1.5TB ram per mode in the cluster

3
SS_MikeOP•

I had a feeling it was Oracle... Thanks!

1
mercenary_sysadmin•

ZFS to present to my small cluster via iSCSI for the VMs to pick up

So is this server a backup target, or is it VM storage? The recommended storage topology varies a lot from one to the other.

The server will have dual 6-core Xeons, 64GB of ECC RAM

Way too much CPU, nowhere near enough RAM. If all this thing does is storage - ie no hypervisor running on the machine itself, etc - a single Xeon e3 is PLENTY of firepower, and then some. Contrary to popular urban legend, ZFS itself is not particularly CPU hungry.

On the other hand, the more RAM, the better, and you're planning on handling a significant amount of storage. It'll work with 64GB, and if your sole use case is as a lightly loaded backup target, that's probably even sufficient. But given that you're clearly throwing money at this box, throw some of it into more RAM.

1
SS_MikeOP•

It will hold VHDs for the cluster and present over iSCSI but they are data only secondary drives to the OSes for DFS namespaces, and just basically file shares for network drives. The primary purpose is acting as a backup server but it does have a secondary role. I have no problem bumping it up to 128GB memory. The CPU is already decided and the server is in hand so I might as well use it. The CPU will see additional use from dedup and archive management processes from the backup software so I'm glad it is overkill.

1
mercenary_sysadmin•

Honestly, I think you're trying too hard to use ALL THE THINGS!

Dedup will most likely murder your performance on this machine. The general rule of thumb for dedup memory planning is 6gb RAM per 1tb of used storage. Assuming two six disk raidz2 vdevs in your pool, that's 32T usable after parity; at 75% capacity that's 24T stored. 24x6 = 144GB, not counting anything that the OS or anything else wants to do besides ARC and dedup tables. It's also not counting RAM used to index that l2arc device you want to deploy.

Dedup is very rarely a win. I've seen it crash performance down to truly awful levels even on machines specced properly to run it. L2arc is also more often a lose than a win; you need an extremely large and extremely hot working set to turn it into a win, which does not sound like anything you're describing here.

What you probably want is lots of RAM, a SLOG device, no l2arc, no dedup, lz4 compression. The big decision you'll need to make is whether you want two raidz2 vdevs, or six mirror vdevs, for your pool topology. I'm a big fan of mirrors, personally, they're easy to manage as well as very high performance.

Last nitpick:

2x300GB 10k mirror for the OS

Why on earth would you spec teeny tiny rust drives here? MLC SSDs will absolutely destroy them for both performance and, more likely than not, reliability, for about the same cost.

2
SS_MikeOP•

The dedup is not file system level, it is application level and performed as a scheduled maintenance task on the backup archives. Thanks for the info. I might go forward with a couple more SSDs and use the 300 10k's as spares for the cluster which already uses them. As far as doing "all the things" - I have the budget to tweak this for performance so I want to spec it out accordingly. There may be instances where I need to use the iSCSI side for more VHDs so I want to be spec'd for more than I need since I can. I don't plan to, but if the need arises I want to know I can handle it in a pinch until I can put another solution in place. I just want to make sure I'm not missing something like the mirror that was mentioned.

1
mercenary_sysadmin•

Just be aware that randomly throwing every possible thing at the system isn't necessarily the path to performance. L2arc will probably decrease performance for the workload you describe - if it has any impact at all.

Glad to hear your dedup is app level.

Be careful to set ashift appropriately for your drives. You almost certainly want compress=lz4, xattr=sa, and I'd recommend atime=off as well.

1
knickfan5745•

Agree with everything but if compression is enabled and you hit the pool hard, you can def utilize a CPU to its full capacity.

1
mercenary_sysadmin•

You have to have a truly fire breathing collection of hardware matched with an insanely anemic CPU for that to happen on lz4.

I manage a sad little A4 APU powered machine that has no trouble whatsoever keeping up with a pair of Samsung 840 SSDs, with lz4 on.

1
knickfan5745•

During our nightly Rsync's to the backup server, the backup server reaches near 100% CPU usage with 2x E5-2609's. Solaris 11.3, 256GB of RAM, ~200TB Pool, LZ4. Perhaps that's because Rsync is also CPU intensive and we run 6 instances simultaneously.

1
mercenary_sysadmin•

That's rsync, not zfs, yes. For that matter, I suspect the majority of it isn't actually 'rsync' either - unless you're using daemon mode transport for your rsync jobs, the majority of that CPU usage is actually going to be the ssh transport for those rsync jobs, which is very punishing indeed with the default ciphers.

1
NastyEbilPiwate•

an SSD for ZIL

You need 2 SSDs in a mirror for your SLOG otherwise you risk data loss.

1
mercenary_sysadmin•

You need 2 SSDs in a mirror for your SLOG otherwise you risk data loss.

Note: you don't risk losing the pool anymore, only whatever transient data is still on the SLOG itself.

2
NastyEbilPiwate•

Yeah I know, I should have been more clear about that.

1
SS_MikeOP•

Thanks. Would you do a hardware mirror or vdev mirror? I have the option for either.

1
mercenary_sysadmin•

Vdev, always. Don't try to layer hardware RAID under ZFS.

This (redundant SLOG vdev) also isn't strictly necessary any longer. Used to be if you lost a SLOG vdev you lost the pool, but that hasn't been true for a while now. Worst case on modern ZFS is you lose whatever's still on the SLOG itself, but the pool remains consistent. That might or might not be acceptable, make your own call.

1
SS_MikeOP•

Right, that would not be an acceptable level of failure so this is good info. Thanks again.

1