secondhand SAS HBA card/expander advice

[Twice_Shy]

So one topic sort of turns into a different topic... but this is good, at least i'm making progress in understanding what I need.


So i'm already totally sold on SAS from this site for awhile, and for the first two NAS boxes I need to build speed is not critical (glorified JBOD bit bucket with SnapRAID) so they can be nearly anything. Yet in the future i'm going to want something that will hopefully saturate faster 5gig and 10gig ethernet connections whether HD or eventual SSD. (and am aware SAS2 should match the first, and the bandwidth of SAS3 is needed to match the second)

What SAS host bust adapter cards and expanders, for both SAS2 and SAS3 are generally recommended or/and available on the secondhand market at a decent price? (ie not $300) I assume hardware RAID modes are supported, but don't know offhand how any of that works if you have more drives than ports due to using expanders/backplanes. Specifically looking for RAID 5 or 6, though i'm not as worried about 6 because the system will be regularily backed up. (wont spend extra for that feature, but I will for RAID 5)

I realize I have more questions than answers and need to learn. I'm strongly leaning towards those Supermicro DF-something cases ie 12-16 bay that I always hear recommended which I believe use backplanes instead of separate expander cards but also need some channels free for an LTO8 drive (I believe they use 4) and a few opticals which would be outside that chassis.

 

[nabsltd]

Yet in the future i'm going to want something that will hopefully saturate faster 5gig and 10gig ethernet connections whether HD or eventual SSD. (and am aware SAS2 should match the first, and the bandwidth of SAS3 is needed to match the second)

If you are talking arrays of spinning disks, then SAS2 is fine to saturate 10Gbps Ethernet. Good SAS disks get about 150-200MB/sec, so somewhere between 5-7 data disks in an array will give you 10Gbps throughput. SAS3 won't help, because no spinning disk can transfer even close to the 500MB/sec of SAS2, much less the 1GB/sec of SAS3. Likewise, there are very few SATA or SAS SSDs that can push much more than 500MB/sec individually, and the ones that can are very expensive.

What SAS host bust adapter cards and expanders, for both SAS2 and SAS3 are generally recommended or/and available on the secondhand market at a decent price? (ie not $300)

I always use SAS3 now simply because the price difference isn't much and you are more likely to have drivers for modern operating systems. You can get many 8-port SAS3 HBAs for less than $100...I bought several Fujitsu CP400i cards for about $80 and flashed them to IT mode for ZFS. It will also do hardware RAID, but I wouldn't use it for that because it doesn't have a battery backup. For less than $200, you can get some form of LSI 9316-8i with battery backup. The IBM M5210 is a good choice, and can be flashed to LSI firmware, although you can get the latest from Lenovo as well. Both of these will work great with an expander backplane.

If you need 16 ports because you don't have an expander, you can get variations of the 9300-16i for less than $150. If you want external connections, I'd advise separate cards, as the -4i4e or -8i8e cards tend to be much more expensive. Just get a boring -4e or -8e card. You can also adapt the internal connection and just route it out through a backplate if you want.

 

[Twice_Shy]

Yeah the SAS2 would be for spinning rust, i'm not going all SSD anytime soon and have plenty of sequential loads where IOPS wont matter like fully mirroring other drives on occasion and acting as a cache for tape archives that haven't yet written to LTO. I'm still fine with just SAS2 for certain cards because I need multiple slower NAS's built anyway for a larger project and saving $50-60 on a card if performance doesn't matter is fine long as drivers work.

I'll make note of the SAS3 recommendations. Can an expander work as part of hardware RAID though? Like if I have an 8 port card, and use an expander to get a few more ports, can I somehow stripe drives on that expander channel or would it force a software RAID mode?

I'm considering cards all the way between 8-24 ports because I thought an LTO drive would take 4 just for itself, and i'm undecided where to stick my opticals - on the NAS box with a 16 drive chassis, or on a smaller 8 drive NAS. (I plan to write multiple discs in parallel of certain data to save time over one at a time) Yet some of those ports can come from the expander if it will work the way I hope.

 

[nabsltd]

Can an expander work as part of hardware RAID though? Like if I have an 8 port card, and use an expander to get a few more ports, can I somehow stripe drives on that expander channel or would it force a software RAID mode?

Expanders are just that...SAS switches that take X lanes of input and switch them to Y lanes of output. Any card that is truly SAS compatible (HBA or RAID) will work with an expander with no special configuration required.

The card sees the drives as "attached", even though they are technically connected to the expander. The card knows this, and will report it, but as far as accessing the disks, there is no difference from being directly attached.

 

[Twice_Shy]

So just to verify whats obvious to someone whose done it but maybe not to me, if the controller supports RAID5 I should be okay to RAID5 any drives that are in the 'network' even if some are direct attached to a controller port, and several others are on an expander... I just have to take total bandwidth bottlenecks at any one point into account, correct? (ie 6gbps per port in SAS2)

I guess I was wondering if say an 8 port RAID card could only stripe 8 drives total regardless of expanders for instance. Or if I put a few expanders onto that and now there's 24 drives, could I be running three stripes of 8 drives or would they be hitting a performance bottleneck since the '8 port' card is only made for and expecting one? (i'm trying to answer the question of whether I can just Expand my way to all the ports I need if I want hardware RAID5 on most of them, vs if I should buy controllers with more ports/consider 2-3 controller cards so that everything is direct attached to an HBA)

 

[nabsltd]

I guess I was wondering if say an 8 port RAID card could only stripe 8 drives total regardless of expanders for instance.

There is no such thing as an "8-port RAID card" any more. There are cards with 8 lanes of SAS2/SAS3 connectivity, and you can then plug those lanes into whatever you want.

If you use breakout cables, then you end up with 2 cables connected to the card and 8 ends that can plug directly into drives.

If you connect 4 lanes to a backplane with one cable and use a breakout cable on the other, you have 4 ends that plug directly into drives and a backplane with some number of slots that drives can be plugged into.

As far as the card is concerned, every drive is connected to the card through a lane of SAS connectivity, whether directly connected or through a backplane. The same lane can be time-domain multiplexed to be used for multiple drives if the drives don't saturate the lane. So, one 12Gbps lane can always handle 2x 6Gbps devices, and likely handle 4-6x in the real world. But, this does mean that no single drive can ever transfer at faster than a single lane of connectivity. So, a really fast 12Gbps SSD will still only give you 6Gbps over an SAS2 connection, even if it is connected to a backplane that uses 4x 6Gbps lanes.

 

[audiophonicz]

Hello, just to "Expander" upon nabs excellent answers:

You'd need like 5x spinning drives writing 225MB/s simultaneously to a single 12G channel to bottleneck it, you'd probably never see it.

Don't confuse Ports (physical connectors) Channels/Lanes (single bandwidth pipe - ie. one 6Gb/s slice) and max drive support. Most LSI HBA/RAID cards support up to 192 drives on a single card. Most HP/IBM/Dell cards are really LSI, and some can be cross-flashed.

An LTO drive doesnt actually use 4 lanes. All the enclosures Ive opened up break out the miniSAS to either a SAS or SATA connector, 2 or 1 lanes respectively, so an internal 300MB/s LTO8 drive would only use 2 lanes.

Unless youre running VMs off SSDs, SAS3 is kinda overkill. A PERC H810 (LSI 9265-8i SAS2) can be had for like $30 including 1GB of replaceable cache and on-board battery backup as opposed to a $120 9300-8i. That paired with cheaper SATA3 drives means you can afford more spindles.
(And maybe even a 2nd 4channel HBA just for your LTO8)

Good luck with your build!

 

[Twice_Shy]

Also thanks to both for helping me understand this better.

There is no such thing as an "8-port RAID card" any more.

Is the full bandwidth of all lanes provided by the card available from any individual port, but also simultaneously shared across ports/aka sort of like an ethernet hub?

Do cards offer more than one lane so that even a 2 lane SAS2 card (2x 6gbps) if set up right with RAID should saturate 10gig ethernet? (also USB 3.2 speed to offload a Samsung T7 external at its max of 1000MB/sec) I'm just not sure I even need SAS3 inside the server if my other attachment point bottlenecks don't exceed 1000MB/sec.

So, one 12Gbps lane can always handle 2x 6Gbps devices, and likely handle 4-6x in the real world. But, this does mean that no single drive can ever transfer at faster than a single lane of connectivity. So, a really fast 12Gbps SSD will still only give you 6Gbps over an SAS2 connection, even if it is connected to a backplane that uses 4x 6Gbps lanes.

Do all backplanes attach with a single port though, and it just electrically decides whether it's using 1/2/4 lanes to that connection? I guess im wondering how lanes are assigned to physical ports with this question.

You'd need like 5x spinning drives writing 225MB/s simultaneously to a single 12G channel to bottleneck it, you'd probably never see it.

Don't confuse Ports (physical connectors) Channels/Lanes (single bandwidth pipe - ie. one 6Gb/s slice) and max drive support. Most LSI HBA/RAID cards support up to 192 drives on a single card. Most HP/IBM/Dell cards are really LSI, and some can be cross-flashed.

If there were two lanes on an SAS2 card (2x6gbps) as long as it was set up right/splitting drives between (if the RAID ability lets it work that way), would it reach the same 12g speed? My highest needed speed was 1000MB/sec one from a USB 3.2 device (samsung T7 at fastest) one from 10gig ethernet in this case.


If HBA/RAID cards support 192 drives, does that mean you could have endless RAID stripes in variations thru multiple links of expanders and such? (like a dozen stripes of 8 drives RAID0 each) I was wondering if there's a performance limitation determined by drives, or just total bandwidth...

An LTO drive doesnt actually use 4 lanes.

Unless youre running VMs off SSDs, SAS3 is kinda overkill.

(And maybe even a 2nd 4channel HBA just for your LTO8)

I don't remember where I saw needing 4 lanes, it was in older notes I took, unless it was a faster tape drive on SAS2 or even SAS1. Would I need a 2nd card just for the LTO8, is that a benefit? Assumes channels = lanes here?

And yeah I kept thinking it was overkill, but i'm fine taking notes for the future for the next jump up in speed that will be here within a few years.

 

[audiophonicz]

Would I need a 2nd card just for the LTO8, is that a benefit?

Youre not going to RAID an LTO, so most RAID adapters wont see/list it.
Most setups have an HBA just for the tape drives
Youre probably not going to fit an LTO inside a chassis with your 12-16 bays, so it will be external, and trying to cable an internal card to an external drive is just messy. Id use a cheap little 1 port external HBA card for the LTO.

If there were two lanes on an SAS2 card (2x6gbps)

No. 6Gbps is 6Gbps. Paths are redundant, not bonded.

does that mean you could have endless RAID stripes

You can have whatever RAID configurations your card and OS supports.

I was wondering if there's a performance limitation determined by drives, or just total bandwidth

There is a performance limitation on all components, all parts of the whole. Every drive, expander, cable, card, bus, filesystem, transfer, protocol, OS, and even physical location determines performance.

Striking a balance between resiliency, capacity, and performance is the name of the game.