most affordable 24pt SAS2 Expander cards? (w/cables, not chassis/backplane)

[Kybber]

I see the Intel SAS expander is mentioned several times, but noone has a vote for the HP expander? It can be found very cheap: I got mine from servershop.de for €25.20. It's chugging happily along with my IBM M1215.
HP SAS Expander Wiki

 

[nthu9280]

I see the Intel SAS expander is mentioned several times, but noone has a vote for the HP expander? It can be found very cheap: I got mine from servershop.de for €25.20. It's chugging happily along with my IBM M1215.
HP SAS Expander Wiki

Added bonus, it also has an external 8088 for connecting tape drive which is one of the asks from the OP. Instructions to update it to 2.10 FW using HBA are in that thread too I think. That will enable support for >2TB

Sent from my Nexus 6 using Tapatalk

 

[Twice_Shy]

I too am curious about the HP but can only go off what other people offer for information because i'm the newbie here. The HP expander was mentioned early in the thread but no additional information or comparisons vs the other options. Reading the linked page of "Overview: The HP SAS Expander is based on the PMC Sierra PM8005 SAS-2 chip which features 36 x 6Gbps ports and 6G/3G multiplexing, SAS 2.0 zoning, self-configuration, table-to-table routing, and an integrated MIPS processor for SES and enclosure management support." I dont even know what those things are! Or if the IBM has them, or the Chenbro, or anything else.

The main things seem to be that HP Expander page suggests that the Chenbro should not be considered, so i'm wondering if the only two options then are HP and the listed Intel, or if there are other makers (or other models from those makers) in the running.

I do prefer the Molex connectored power supply option of the Intel RES2SV240 but that's not a hard rule, it's more if the money is similar because I wanted to have the box of drives separate from the motherboard running them and it avoids having a second "basically just there to power a card" mobo in there. (unless there's some kind of adapter or modification to use direct power on a pcie card like that)

The dual linking is of interest to me, since it sounds like a dual linked card should enable 12gbps total throughput letting me saturate 10gigE in the future? That's a major plus/no upgrading just for speed, just reconfiguring for RAID-like vdev stripes in the future under ZFS. (though it will start under SnapRAID and mere 1gigE, only need a cheap bit bucket not a performance array at the beginning and for the first system - I could use different hardware in each but ideally i'd find one that works for all future plans and just standardize around that to ease troubleshooting)


Questions I still dont understand at this point:
- Someone mentions the 24port expander only supports 20 drives, and the 36 port expander only supports 24 drives (??)
- How to add an external SAS port if I use the internal-only expander (and what that costs if nontrivial)
- Performance comparison of the "triple HBA option" vs Expander option, since i'm using spinning rust not SSD's anyways even saturating 10gigE should be doable on a pair of 6gbps ports I assumed if the drives can keep up
- "If you are using SATA drives I suggest additional controller ports instead of a SAS expander. SAS expanders do the SAS / SATA conversion for attached SATA drives, which hides things from the controller (and thus from the host OS)." - how much of a concern this is, ie what is hidden and why would it be/thought everything just was directly accessible to the host anyway/where does this become a problem

 

[i386]

@Twice_Shy
One link = 4x 6gbit/s -> 24 gbit/s,
dual link = 2x (4x 6gbit/s) -> 48 gbit/s
One link can be enough to saturate 10 gbe

Someone mentions the 24port expander only supports 20 drives, and the 36 port expander only supports 24 drives (??)

You need a link between raid controller/hba and the expander. The 36 port has two external SFF-8088 ports (=8 sas/sata ports) that can be used to daisychain jbod chassis

How to add an external SAS port if I use the internal-only expander

There are adapters for the pci slots which have sff-8087 internal and sff-8088 external, google says the start at $40

Performance comparison of the "triple HBA option" vs Expander option

If you want local storage than more hbas would give you more bandwidth, shared storage will be bottlenecked by your network (unless you use 56+ gbit/s infiniband/ethernet)

 

[Kybber]

I do prefer the Molex connectored power supply option of the Intel RES2SV240 but that's not a hard rule, it's more if the money is similar because I wanted to have the box of drives separate from the motherboard running them and it avoids having a second "basically just there to power a card" mobo in there. (unless there's some kind of adapter or modification to use direct power on a pcie card like that)

On page 3 of the HP SAS Expander wiki thread, you'll see an adapter that is reported to work.

 

[Twice_Shy]

@Twice_Shy
One link = 4x 6gbit/s -> 24 gbit/s,
dual link = 2x (4x 6gbit/s) -> 48 gbit/s
One link can be enough to saturate 10 gbe

You need a link between raid controller/hba and the expander. The 36 port has two external SFF-8088 ports (=8 sas/sata ports) that can be used to daisychain jbod chassis

There are adapters for the pci slots which have sff-8087 internal and sff-8088 external, google says the start at $40

If you want local storage than more hbas would give you more bandwidth, shared storage will be bottlenecked by your network (unless you use 56+ gbit/s infiniband/ethernet)

So wait, did I misunderstand? I was assuming that one "link" was a single 8087(?) line of the 6gbps, so that for instance the Intel RES-whatever card which seemed to be was splitting one link Expanded into 24 links (via 6 of those 'quad' outputs, though I hadn't yet figured out where the input port was) to support 24 drives via a single input channel. So that an HBA with two jacks and technically 8 channels of output was using 1 of the 8 (to run 24 drives) and would have 7 drive connections leftover and here I was thinking 24+7 would be enough to run my 16-32 (well 30, +external tape) range.

I'm pretty sure I am now wrong on multiple assumptions but then i've openly acknowledged I dont understand how it all works. Can you walk me through what the wiring would technically be? Assume one "8 port" HBA, and two 24 port SAS Expanders daisy chained one to the next (even if I could hook both direct) just so I get a better mental model of how this fits together/how many ports I lose where/how fast the links are.

Again quickie version - mostly planning 16-32 drives, eventually saturating 10gig ethernet, but the ability to run more drives a future possibility/say worst case 64 drives? (biggest possible array I can contemplate)

On page 3 of the HP SAS Expander wiki thread, you'll see an adapter that is reported to work.

That looks like a super-useful adapter, thank you for the reminder. I hadn't read past 2nd page of the link since nobody else had mentioned the HP as recommended cept the first response originally. Looking through the rest of the thread I didn't see any mention of where it was bought/where to get one myself though.

 

[markarr]

So wait, did I misunderstand? I was assuming that one "link" was a single 8087(?) line of the 6gbps, so that for instance the Intel RES-whatever card which seemed to be was splitting one link Expanded into 24 links (via 6 of those 'quad' outputs, though I hadn't yet figured out where the input port was) to support 24 drives via a single input channel. So that an HBA with two jacks and technically 8 channels of output was using 1 of the 8 (to run 24 drives) and would have 7 drive connections leftover and here I was thinking 24+7 would be enough to run my 16-32 (well 30, +external tape) range.

I'm pretty sure I am now wrong on multiple assumptions but then i've openly acknowledged I dont understand how it all works. Can you walk me through what the wiring would technically be? Assume one "8 port" HBA, and two 24 port SAS Expanders daisy chained one to the next (even if I could hook both direct) just so I get a better mental model of how this fits together/how many ports I lose where/how fast the links are.

So an 8 port sas hba typically divided into two sff-8087 ports so each port is 4 of the 8 channels, which means each 8087 port has 24gbps sas throughput and supports 4 drives. Now when you attach it to an expander you are trading the 4 ports from the hba port for the 20 on the intel expander for a total of 24 sas ports that can accept drives between the hba and the expander. The cable that goes from hba to expander is a sff 8087 cable that carries 4 channels and you cannot split that out. The intel expander has a total of 24 channels on it divided up into 6 physical ports. You use 1 of the 6 physical ports off the expander and 1 of the 2 physical ports off the hba to connect the two together to give you the aforementioned 24 ports total between them.

You can hook two expanders to the hba you would run a cable from each port of the hba to the two expanders, which would give you 40 ports total.

 

[Twice_Shy]

Hmm okay, so even the '3gbps' speed SAS1 Expanders, if only using 2tb drives (or somehow compatible with beyond 2tb) could have saturated 10gigE because it forces it to use 1 physical 8088/4 ports of 3gig data/total 12gig dataspeed. (which I guess is another reason SAS1 expanders arent off the table IF anyone has experience or knowledge of having them work with over 2tb drives since it sounds like some do but i cant find more specific info)

What about daisy chaining? If I wanted more than two expanders/40 ports, i'm supposed to be able to daisy chain... is that supported by all expanders? Only some? Is it as simple as just running one physical/4 ports to yet another card or not? Also curious about daisy chaining 'internally' (two expanders in one physical case) vs externally (creating an external port to run to another chassis full of drives) if its different which I assume it is.

Also if you know, how do I turn an internal port into an external port for the tape drive? (on an expander lacking an external port) Does that split one of the four ports off somehow, or take an entire physical (4 ports worth) of data or what will it normally be for an Ultrium LTO6/7 tape drive?

 

[markarr]

Hmm okay, so even the '3gbps' speed SAS1 Expanders, if only using 2tb drives (or somehow compatible with beyond 2tb) could have saturated 10gigE because it forces it to use 1 physical 8088/4 ports of 3gig data/total 12gig dataspeed. (which I guess is another reason SAS1 expanders arent off the table IF anyone has experience or knowledge of having them work with over 2tb drives since it sounds like some do but i cant find more specific info)

What about daisy chaining? If I wanted more than two expanders/40 ports, i'm supposed to be able to daisy chain... is that supported by all expanders? Only some? Is it as simple as just running one physical/4 ports to yet another card or not? Also curious about daisy chaining 'internally' (two expanders in one physical case) vs externally (creating an external port to run to another chassis full of drives) if its different which I assume it is.

Also if you know, how do I turn an internal port into an external port for the tape drive? (on an expander lacking an external port) Does that split one of the four ports off somehow, or take an entire physical (4 ports worth) of data or what will it normally be for an Ultrium LTO6/7 tape drive?

I dont know if a SATA drive on SAS 1 is 3gbps or 1.5gbps, it depends some expanders you can upgrade the fw to allow >2tb but have not seen much success. So if you want to use bigger than 2TB drives just go with SAS2 you will save yourself headaches.

You can daisy chain expanders. It does not matter if its internal or external they work the same way. The only difference is the physical connector the underlying SAS is the same.

You can there are adapters that will convert sff-8087 sff-8088 to take internal to external and vice versa. That is a physical port so it will take 4 ports worth.

 

[Twice_Shy]

@Twice_Shy
You need a link between raid controller/hba and the expander. The 36 port has two external SFF-8088 ports (=8 sas/sata ports) that can be used to daisychain jbod chassis

There are adapters for the pci slots which have sff-8087 internal and sff-8088 external, google says the start at $40

If you want local storage than more hbas would give you more bandwidth, shared storage will be bottlenecked by your network (unless you use 56+ gbit/s infiniband/ethernet)

Okay, so basically then, if I want a separate drive chassis from where I have the motherboard (and I do) i'll be using an SFF-8088 port to connect HBA to drive chassis. The drive chassis needs to have two SFF-8088 ports if I want to daisy chain to a second.

If each SFF-8088 port is worth about $40 that definately nudges me to prefer expanders which would have a built in 8088 port or two. Since i'm seeing external cables are more expensive than I originally hoped I may reconsider my preference for separate drive chassis (will plan to do that upgrade later) since again the goal for all this is still minimum total overhead cost per SATA drive.

Is a normal Ultrium LTO SAS drive connected via a SFF-8088 wire? (whether it needs it or not, does it suck up basically 4 ports) I assume it shouldn't matter where it's hooked - right off the HBA vs on the far side of an SAS Expander to daisy chained SAS Expander chained to the LTO Ultrium drive.



Everything i'll be using is spinning rust for now so not too worried about absolute bandwidth - I would like to eventually play with 40-56G Infiniband but that's a ways off in the future and probably two rounds of upgrades before i'm even looking at it and not before a bunch of SSD's is afforded too.


EDITS: Since i'm re-perusing that HP Expander 36 port model, and it says you can dual-link the bandwidth (8 ports total), what happens with daisy chaining/can you dual-link twin ports beyond that as well? I suppose that would drop it to only 20 drives then if performance critical. (8 ports in from the HBA, 8 ports out to the next Expander)

Does anyone know if the Intel RES2SV240 can dual link?

 

[sfbayzfs]

I highly recommend an LSI SAS2 HBA SAS2008 or SAS2308 coupled with either one or more HP or Intel SAS2 expanders (if you can't use a Supermicro chassis with a built in SAS2 expander in the backplane(s), that is.)

I (and various local friends) have had great luck with both the Intel and HP SAS2 expanders - the HP needs PCIe power, but gives you more total ports and an external port, the Intel is low profile compatible and can use PCIe or Molex power. For years I ran a 20-disk zpool from an M1015 single cabled to an Intel RES2SV240 expander, and the 20 drives hanging off the other 5 ports, and there were still 4 unused ports on the controller, a good trick for handling 24 drives with one HBA and the Intel expander. My friend duplicated the setup except with the controller dual linked to an HP SAS2 expander, and it has been rock solid as well. These were in tower cases, if you do that, make sure that you have a side fan on your case that blows a decent amount of air on the heatsinks of both the SAS2 expander and the SAS2 controller.