Half height HBA with 16 or more ports?

[Brandon_K]

Hey all.

I'm putting a new Unraid box together and I've made some changed along the way that has compounded some things for me.

The new build is based off of a i5 12600K. Unfortunately due to consumer motherboard limitations, I'm stuck with only two PCI 4.0 x16 slots and a PCI 5.0 x16 slot. My plan is to fill those with a 5gb 1060, dual port Mellanox 10gbe (HP branded) and a HBA. The on-board M.2 slots will be filled with four 500gb SN750's, one pool for cache, one for docker and VM's.

Initially, I was planning on running only my two external shelfs, a DS2246 (24x2.5) and a EMC KTN-STL3 (15x3.5).

But... I've since stumbled across a pair of SM chassis that I'll end up using one of to house them in. One chassis is a 822T-400LPB w/ 6x3.5" LFF, SATA backplane. The other is a 826, but that's all I know about it right now, I bought it with next to no detail, other than it's 12x3.5 LFF, plus the 2x2.5 rear option. I have no idea if the backplane on it is SATA or SFF, but either way, I'm assuming I'm going to need more ports on the HBA.

Initially I was going to try to find something like a 8i8e or 16/24i, then use a adapter to pull the internal ports external to run the external shelfs, but from my bit of searching, it doesn't look like anything is available in a half height card that will fit in a 2U chassis.

Does anyone have any suggestions? Port expanders maybe? I have zero experience with them. While I have high speed storage via the NVME drives, I'd like to keep reasonable throughput through the HBA. I'm open to any suggestions you guys may have!

 

[BlueFox]

Most of the newer Broadcom cards do in fact have low profile options on the 16 port versions. 9305-16i for example. Same for newer generations.

 

[itronin]

The 826 is LFF. the 216 (2U) is SFF.

The 826 has quite a few backplane options.

they're also relatively easy to switch out.
types, TQ = SAS(1-3)/SATA direct with SATA connector, A = SAS(1-3) Direct with SFF-8087 connector, EL1 = 1 expander onboard, there's also a hybrid with up to 4 NVME (8643) and up to 12 direct SAS/SATA via 2 8643 in a split between the two.

Depending on your IO you could probably get away with an -8i controller.

the BPN-SAS3-826EL1 is generally < 100 in the US on the bay.
It has 4 SFF-8643, you could do one cable to that from your -8i. Chain the rear 2.5" off another backplane 8643 and then feed one 8643 from your HBA to a HH 8643 to 8644 bracket or an SFF-8087 to SFF-8088 bracket to chain your shelves or (and use an SFF-8643 to SFF-8087 cable from your HBA)...

with a 16i you can do more interesting things. One shelf could have 8 lanes if you had hot IO there. Lotsa options.

You'll just have to watch your cable lengths I think with adapters and inside the chassis cabling.

IMO the 826 is a better choice than the 822T

 

[Brandon_K]

Most of the newer Broadcom cards do in fact have low profile options on the 16 port versions. 9305-16i for example. Same for newer generations.

I don't have any first hand experience, but I've been under the impression that 3Ware and Broadcom branded HBA's do not play well under Unraid, so I've been steering clear of them. Not to mention the cost.

 

[BlueFox]

Broadcom is kinda the industry standard. I'm not sure what you'd have in mind for alternatives? There really aren't any other large vendors left for HBAs.

 

[Brandon_K]

types, TQ = SAS(1-3)/SATA direct with SATA connector, A = SAS(1-3) Direct with SFF-8087 connector, EL1 = 1 expander onboard, there's also a hybrid with up to 4 NVME (8643) and up to 12 direct SAS/SATA via 2 8643 in a split between the two.

Depending on your IO you could probably get away with an -8i controller.

the BPN-SAS3-826EL1 is generally < 100 in the US on the bay.
It has 4 SFF-8643, you could do one cable to that from your -8i. Chain the rear 2.5" off another backplane 8643 and then feed one 8643 from your HBA to a HH 8643 to 8644 bracket or an SFF-8087 to SFF-8088 bracket to chain your shelves or (and use an SFF-8643 to SFF-8087 cable from your HBA)...

Excuse my newb'ness here.

Are you saying with the 826EL1 that has four SFF-8643's, that I would only need to connect a single cable from a HBA to it to have all 12 drives running? I could then use the backplane as a "splitter" to feed off the two 2.5" drives in the rear? So with a -8i HBA, one to the internal drives, then one to feed the external shelfs? or with a -16i, I could send two up to the 826EL1, still split one back to the 2.5" drives, then each external shelf would get their own channels? And this is because it's an expander?

And hypothetically just because I'm sitting on a 9207-8i, could I go SFF-8087from the HBA to SFF-8643 to the backplane? I see that said cables do exist, but that doesn't confirm to me that it actually works. Then I could loop the second port to the 8087/8088 adapter out to the shelfs.

 

[ericloewe]

Are you saying with the 826EL1 that has four SFF-8643's, that I would only need to connect a single cable from a HBA to it to have all 12 drives running?

Yes. Two cables for extra bandwidth, too (restrictions apply, but that's the concept).

I could then use the backplane as a "splitter" to feed off the two 2.5" drives in the rear?

Yes. Or feed the next expander (internal in a larger chassis or external)

So with a -8i HBA, one to the internal drives, then one to feed the external shelfs?

Sure, or get a -4i4e or similar card.

or with a -16i, I could send two up to the 826EL1, still split one back to the 2.5" drives, then each external shelf would get their own channels?

Sure, I guess.

And this is because it's an expander?

Well, that's what's going on in the external shelves, too. Think of expanders as SAS switches.

And hypothetically just because I'm sitting on a 9207-8i, could I go SFF-8087from the HBA to SFF-8643 to the backplane?

Sure, the only catch, if you can call it that, is that older drivers/firmware were very buggy with mixed SAS2/SAS3 scenarios. This should be resolved by now.

 

[itronin]

... or with a -16i, I could send two up to the 826EL1, still split one back to the 2.5" drives, then each external shelf would get their own channels? And this is because it's an expander?

There's a chance (probably better than even money) that your 826 may come with the BPN-SAS2-826EL1. That will have two SFF-8087. Using your existing HBA you could connect 1 SFF-8087 to it and bring the other out the back of your chassis for your shelves.. Connect two SATA ports to your rear 2.5" bays.

And hypothetically just because I'm sitting on a 9207-8i, could I go SFF-8087from the HBA to SFF-8643 to the backplane? I see that said cables do exist, but that doesn't confirm to me that it actually works. Then I could loop the second port to the 8087/8088 adapter out to the shelfs.

If you have not done so please do update your HBA to the latest P20 firmware. If the card is an OEM card then cross flashing may be required - YMMV.

 

[ericloewe]

P20.insert_zeros.07. Yes, it makes a difference. Early P20 was a trainwreck. 05 was mostly stable, but 07 is most stable.

 

[Brandon_K]

Thank you for all of the help and education guys!

There's a chance (probably better than even money) that your 826 may come with the BPN-SAS2-826EL1. That will have two SFF-8087. Using your existing HBA you could connect 1 SFF-8087 to it and bring the other out the back of your chassis for your shelves.. Connect two SATA ports to your rear 2.5" bays.

If you have not done so please do update your HBA to the latest P20 firmware. If the card is an OEM card then cross flashing may be required - YMMV.

I finally got some pics from the seller today. It is indeed a SAS2-826EL1 backplane. I paid $200 for the server, which is the 826 chassis, (4) 3TB WD Re drives, X9DRD w/ two 2603 v2's, 24GB, etc. I figure I can sell off the parts that I don't plan on using which will offset the cost of the SAS3 backplane.

 

[itronin]

Thank you for all of the help and education guys!



I finally got some pics from the seller today. It is indeed a SAS2-826EL1 backplane. I paid $200 for the server, which is the 826 chassis, (4) 3TB WD Re drives, X9DRD w/ two 2603 v2's, 24GB, etc. I figure I can sell off the parts that I don't plan on using which will offset the cost of the SAS3 backplane.

You're running unraid right? Why not use that motherboard which has an onboard LSI 2308 and SAS, the slots bifurcate which makes for LOTS of enterprise nvme options. You can put your HBA in there too and dedicate it to your shelves with an internal to external bracket.
You can put tons of relatively inexpensive DDR3 in that server too...

No mucking about switching motherboards (which you'll need a cable adapter to go from the SM ribbon cable to your consumer board if you do switch).

pop your consumer board into a low cost chassis,run Linux, plex, emby jellyfin, whatever you want and just serve up the filesystem to your media server box. Heck you could even put in a low cost hpe 40Gbe IB/Ethernet card into the storage server and your media server and give them a nice speedy piple using a low cost QSFP to QSFP cable (like netapp or something). (not that you'd us all that bandwidth though .)

@Brandon_K you got a good deal on that server btw...

 

[Brandon_K]

You're running unraid right? Why not use that motherboard which has an onboard LSI 2308 and SAS, the slots bifurcate which makes for LOTS of enterprise nvme options. You can put your HBA in there too and dedicate it to your shelves with an internal to external bracket.
You can put tons of relatively inexpensive DDR3 in that server too...

No mucking about switching motherboards (which you'll need a cable adapter to go from the SM ribbon cable to your consumer board if you do switch).

pop your consumer board into a low cost chassis,run Linux, plex, emby jellyfin, whatever you want and just serve up the filesystem to your media server box. Heck you could even put in a low cost hpe 40Gbe IB/Ethernet card into the storage server and your media server and give them a nice speedy piple using a low cost QSFP to QSFP cable (like netapp or something). (not that you'd us all that bandwidth though .)

@Brandon_K you got a good deal on that server btw...

Mostly because I'm trying to pair back on machines in the house, as well as the power that they require. I'll be going from machines that consume more power at idle (DL380 G9, another SM build single Xeon box, a 10th gen i5, etc), than this i5 12th gen does at full tilt and I'll still have more processing power and speed. Just my current UPS (SMT2000RM2U) pulls 100w at idle. That is getting replaced by a modified SMT1500 with four external TX30 batteries. More efficient, more runtime, less idle draw.

Not to mention better workflow when it comes to keeping all of the containers on a single system. So, less kwh power, less headache, more processing power, plus QuickSync (for now). Rough numbers show I'll be able to sell off the single Xeon Supermicro box, DL380, etc for right around the total cost I'll have in to building current tech in a single box.

 

[itronin]

Understood.

So this wasn't lost from my previous comment:

You will very likely need to either fabricate something like this supermicro fp cable adapter or buy one from SM or AMZ, or the bay. NB: supermicro shipping when combined with the price of an item is often more than the "higher cost" listing that includes shipping. If you need multiple parts from SM then it can be cheaper to buy from them. YMMV.