Optimizing/Configuring ZFS based SMB
Background:
Tech enthusiast who does something VERY different for a living. Smart but not highly competent in this domain (read: better than 99% of randoms on the street, worse than 90% of people doing IT in a role beyond help-desk, previously worked with a bunch of network eng at an F500 but in a different capacity). I have a handful of issues and in some cases I don't know the NAME of the solution I should be asking for.
I wanted more performance/flexibility out of my NAS so I built out the following using mostly spare parts:
Hardware:
HPE Proliant Gen 10 (Excavator based Opteron x3421 2M/4T)
32GB non-ECC RAM - everything TRULY critical will also be backed up to a cold disk and stored off site
4x 6TB CMR Harddrives
64GB Flash drive with OS (logging is currently still on this)
System only supports 4x 3.5" HDD. Some space for a 2.5" SSD and 2AIC slots (PCIe 3.0 x4 + x1)
Current config:
Ubuntu 20.04 LTS Server with ubuntu-desktop GUI installed
1x pool RAIDZ1
SMB for sharing on network, configured via point/click GUI
1 open share (no login needed), 3 non-open shares (STEAM, backup, misc files)
Currently ~200GB of data stored on NAS
1Gbps networking.
Future purchase considerations:
Optane Gen 2 for L2ARC? (not currently released; should have solid performance)
Better drive for OS?
mGigE?
Goals:
Have photo albums be VERY responsive.
Have Windows_file_history backups be reliable enough.
Have games eventually be able to be handled via ISCI with the host caching via an SSD.
Some issues:
Occasional 1-5s lag when first accessing a large folder within a share that has many photo thumbnails. After that it feels almost as responsive as a local SSD. Could this be the system waking up from sleep? USB flashdrive going crazy? Lack of RAM caching? Should I be looking into L2ARC?
NAS doesn't show up on the network by default. Connecting to it requires me to map shares. Would like to show \NAS_name on network.
Posted Jun 29, 2020, 1:22 AM UTC · edited
The ARC only only caches data that has been accessed before. So if you only rarely look at the folder, no form of caching will help your thumbnails performance
A 'special vdev' for meta data and small files may be your best bet. What size is the typical windows thumbnails file? ZFS allows you to mark a mirror of fast SSD's as where all the small files (<128kb) will be stored. Increasing random access.
These type of accesses hdd really such at, look at 4k random stats in Crystal disk mark, like 50 iops. A good SSD can do 10k such iops.
But do check the 4k random performance of what you are going to buy. These numbers aren't nearly as sexy as the "3000 mb\s" the marketing people like, so they are hard to find.
Yes, you need at least a mirror. As the SSD is storing the map of where all the files are, if your 'special vdev' dies, you lose everything.
I've opened the folders before so this is a recurring thing. With that said open -> stutter -> close -> open -> no stutter seems to be common. For small folders it's seemingly instant.
I suspect that thumbnails are all small. No idea what their size is. It's whatever is dynamically generated, either by windows or the NAS (no idea what the mechanism is). Windows box has 64GB RAM, a 16 core Ryzen 3950 CPU and is all flash based storage.
I take crystaldiskmark with a grain of salt but testing with 1GB of data - ~100 OPS on seq1M, 25000 and 1500 for RND4K depending on queue depth when looking at IOPS. Read speeds are ~100MBps on all CDM tests except 4kqs1, which is more like 6MBps.
FWIW I prefer the idea of replicative caching - as in more copies of the data being made over the idea of a special vdev - partially because I want to be able to assume that SOMETHING will die and that I'll need to port this over in the near to moderate future. I don't want to assume my own competence.