SAS - HBAs, Ports, and Expanders - oh my
I'm hoping somebody can clear up some confusion here. I've built out a Gluster on ZFS cluster on SuperMicro hardware using LSI 9207-4i4e HBAs running over an SFF 8087 to an internal 12-disk expander with 4 TB SATA drives. We initially purchased the 4i4e version of the 9207 to support JBOD expansion down the road, which is where I'm at now.
I've been in conversations with our vendor regarding the expansion, and they seem to think that putting a 24-disk JBOD on the external SFF 8088 port on the 9207 will cause performance issues for the internal 12-disk expander. In their words, any time you attach another expander backplane to the same HBA, performance will degrade. Their recommendation is to add an HBA for every expander we add (cough).
This doesn't make sense to me, because each expander will be on it's own dedicated 4-lane SAS 2.0 port, for a theoretical 24 Gbit/s of throughput. My thinking is that the 24-disk expander could become oversubscribed in some edge cases, for example all 24 disks performing streaming reads at ~133MB/s (7200 RPM SATA) comes to ~3.2GB/s, which is slightly more than that 24Gbit/3GB per second of bandwidth the 4-lane SAS connection can provide. I can live with that, as long as the performance of the other 12-disk expander isn't affected.
I think my vendor is confusing the addition of expanders to the same 4-lane port (cascading) with putting different expanders on different ports. However this is the first time I've configured a SAS JBOD expansion, and I want to make sure I understand the protocol before making a purchase. If I do need to add another HBA, I'll probably just go with an LSI 9300 which is SAS 3.0, and can provide plenty of overhead bandwidth for the 24-disk expansion instead of being oversubscribed.
Thanks for reading!
Posted Nov 26, 2014, 9:46 PM UTC
It sounds fine to me, except for putting SATA drives behind an expander: some people have reported that causes issues with ZFS. I haven't tested it personally.
Edit: Nexenta says it's "proven conclusively to be highly toxic".
http://garrett.damore.org/2010/08/why-sas-sata-is-not-such-great-idea.html
Running SATA behind an expander on ZFS (and Nexenta), can confirm is toxic.
Some tweaks and we're running OK, performance is kind of cruddy (we had drives get dropped by ZFS due to not responding in less than 2000ms, even though this event is rare) but it's archival.
Interesting, what sort of toxic behavior were you seeing? Was the drive latency issue reported in syslog, or did you have to instrument that?
Mainly just really high latency spikes so far...
Nexenta's drive health system would drop the drives due to them taking >2000ms to respond when I got on 4.0 (has better failure detection). I disabled that functionality for now until we discuss replacing them with SAS (I wanted SAS in the first place but got shot down for the marginal extra cost).
It's controversial, in that some companies have pulled it off, like Backblaze maybe?
Regardless, with the cost difference per drive being only $20, OP should get SAS 4TB drives instead of the SATA variant and skip any future headaches.
I didn't realize that SAS disks had come down in price so drastically, so I'll definitely be looking into that. The drives we currently use are Seagate "Enterprise" SATA, so I'm not so much worried about a difference in build quality between them and 7200RPM SAS, but the extra protocol complexity is definitely a valid point.
FWIW, Nexenta is also based on Illumos, whereas I'm using ZFS on Linux (Centos 6). It wouldn't surprise me if there is a quality difference between the mpt driver for illumos vs. Linux.
Anecdotally, we actually had a problem recently when we upgraded to firmware P20 on the LSI 9207. We were getting I/O errors from the mpt driver in syslog for a week before anybody noticed - turned out there was a bug with the P20 firmware affecting communication with the expander backplane.
ZFS just kept doing it's thing, silently correcting errors and moving on. We did eventually start to get latency issues on one of the three boxes in this state, realized what was going on, and downgraded back to the previous firmware. A ZFS resilver and a few gigs worth (out of 60+ TB) of rebuilt data later things were back to normal. ZFS really saved out butts - I hesitate to imagine what would've happened on hardware RAID and XFS/ext4.