r/freenas•
telim

Best use for 2x NVME Drives

Hi all, looking for some opinions about the proper use for 2x 480gb high-speed NVME drives (Corsair 510 series) in a FreeNas setup.

I'm using two Kingston 120GB SSDs that I got on special as mirrored boot drives.

I'm using 8x 10TB Western Digital White-Label drives shucked from external cases in a ZF2 array (~50TB available).

Main use case: Media server for TV Shows / Movies. ~1TB of "mission critical" household data (Pics, home movies, etc.).

I intend to install Ubuntu as a VM on the same machine to serve Plex and all the supporting apps for that, torrenting, etc.

Obvious options:

-SLOG / write cache (mirrored, striped, or just 1 used for this; know that these aren't ideal as no battery)

-L2ARC (likely no role for this, as most data will be requested from the NAS only once/twice)

-Metadata drive for PLEX - likely the ideal usage?

-Transcoding drive for plex (not ideal to use a SSD of any kind for this, would prefer to somehow use some sort of ram-disk for this use, think you can use the /tmp drive in Ubuntu is a pseudo-ram drive)

Does anyone have any thoughts? Once my media is transferred to this box I likely wont be doing many large transfers to it.

Appreciate the input!

Also, If anyone has a good resource for Freenas 11.2 u6 and installing Ubuntu, I'm all ears!

Posted Oct 8, 2019, 1:41 AM UTC

914
rtznprmpftl•

I would create a pool with them to store the Plex metadata and the VM image.

8
VIDGuide•

Yeah. My SSDs on my recent build weren’t that good (SATA 240gb), but that’s what I did. Built a separate pool of those 2, and my VMs and application storage, including plex, all live on here. Works great.

2
telimOP•

Mirrored or striped?

2
joshuata•

I would recommend mirrored. Striped wouldn’t gain you too much performance since nvme is already pushing limits, and mirrored would give you useful redundancy

3
rtznprmpftl•

Hmm, I would just backup everything on the main raid every day and go full YOLO.

But, is also use more experimental stuff like bcachefs so I know that my backups work :D

1
BabaJega1911•

It depends on the rest of your system and the amount of ram you use.

If you use 16gb of ram, adding ZIL / L2ARC could lower your performance.

If you use (for example) 128GB of ram, adding ZIL / L2ARC would most likely increase your performance.

3
humbleservant2992•

Can you elaborate on this a bit more please? I have 32gb of ram purchased for my build. Was planning on using nvme drives for l2arc/zil. The more I read the more I’m thinking that the ssd drives for cache was a bad idea. I just don’t why? And why would it be more beneficial with 128gb of ram vs 16? Is there somewhere the documentation that covers all this?

2
BabaJega1911•

The reason for this is that for caching devices an index is created and placed in the ram. The bigger the drive, the bigger the index, the more ram you need, the less ram you have available for your ARC (main zfs cache). When your ARC is reduced but you have a large L2ARC instead, overall perfromance suffers since the ARC is way faster than the L2ARC.

5
lwwz•

ZFS makes excellent use of RAM to cache volumes. You're almost always better off going with more RAM.

3
cw823•

This. Most people will not benefit from slog or l2arc, and them a subset of those people that would benefit will use devices that don’t perform well for those purposes.

An example would be consumer SSDs for slog.

3
humbleservant2992•

but what if I have 12TB of slower 5400-7200RPM disks . and of that 12TB, 2-4tb of it is constantly being accessed as esx datastore, Plex & Nextcloud... my 32gb of ram will run out real quick. Which is where I would think the much faster NVMe SSD's would come into play.. no??

Do other NAS software handle SSD cacheing significantly different then FreeNAS? I recall back in 2011 my employer bought a pre-built Open-E NAS and had an SSD cache in it. The way they explained it to me was that it would start to cache the most commonly used data read/written until the SSD filled up and then replace with whatever data needed to be cached more.

1
telimOP•

64gb of Ddr4 ecc ram

1
Mizerka•

hey got similar setup on unraid;

you're probably best using them for slog in raid1(btrfs).

metadata storage is also a good choice but... I found little improvement from that. Big improvement was using ram disk like you said, by forcing it to run in /tmp path. I also use ssd's for dockers and vm image

1