Supermicro BPN-SAS3-216A-N4 cabling questions...got some answers from Supermicro.

[jlficken]

I'm trying to figure out if all of the SFF-8643 connectors are required (ie I need 3xHBA's) for this backplane or if it's like a SAS2 backplane where only 1 is absolutely required and the rest are for redundancy and external storage?

I've read the manual but I don't see it spelled out anywhere.

I won't be using the NVMe connectors but will use all 24 bays connected via SAS3.

https://www.supermicro.com/manuals/other/BPN-SAS3-216A-N4.pdf

 

[jlficken]

I decided to hop on chat with Supermicro through their store as they appear to have a "technical support" option.

I have another server with this backplane:

https://www.supermicro.com/manuals/other/BPN-SAS3-216EL.pdf

The difference is that the BPN-SAS3-216A-N4 is *not* an expander backplane so each SFF-8643 cable is responsible for 4 drives. I wondered about that since 24/4 = 6 cables which is what I see in my server.

If I wanted to use the last 4 bays with NVMe drives I'd want to get something like the AOC-SLG3-4E4T in order to attach 1 cable to each of the 4 NVMe ports on the backplane.

At some point my plan will then be to grab 2 of the AOC-S3008L-L8E and 1 of the AOC-SLG3-4E4T HBA's to turn this into a flash based TrueNAS Core server for shared storage with a Proxmox cluster with the NVMe drives being used for L2ARC. I've never actually used TrueNAS/FreeNAS beyond playing with it in a VM so it should be fun.

Now I know

 

[BlueFox]

EL = expander. A = direct attached multi-lane connectors.

 

[i386]

EL = expander. A = direct attached multi-lane connectors.

TQ = direct attach like A backplane, but every port has it's own cable (pretty messy )

 

[itronin]

As @i386 said TQ is pretty messy but it is also still pretty congested with the 216A-N4. I have one in service as a TNC storage server for my hyps. Total of 9 8643 - 8643 cables. 5 for the SAS SSD's (4 lanes per cable) and I have a pair of dual u.2 to pcie x8 cards (basically this ) and slot bifurcation on an X10SRL-F. One 9400-16i, 1 AOC3008 with just 1 8643 cable (in an x4 slot). With the real dual 2.5 I could conceivably have 22 SAS drives using a break-out cable coming off the unused 8643 on the 3008.

I also have a server with an 826A-N4 - same deal just fewer cables.

 

[ano]

Well, this is confusing, I have some BPN-SAS3-216A-N4 incoming

The SAS3 part is ok, connected to 9500's I have that allready in prod

I have H12ssl-i, which gives me a single nvme onboard, which I think I can connect to 2 out of 4 connectors. with CBL-SAST-0826, will I be able to connect gen4 devs at gen4? speed? will I have trouble with gen4? obvious choice would have been to get the 2 connector h12... but the price was right on the single, or we will see...

now what do I do with remaining two port there? can I use them for sas/sata drives? if so.. how what would be cheapest way? (got a lot of lab'ish systems to build that might see prod if all goes well, so cant ghetto it too much (ceph)

AOC-SLG4 cards seemto be very hard to get, in theory you could use CBL-SAST-0826 as well, to get gen...4? again

now.. AOC-SLG3 cards are very availble, but which cable would you get for the white SFF8643, as the other end is also a 8643, guessing this? CBL-SAST-0658? kinda meh, but I would be able to use optane at least, and get gen3 speeds... for gen4 devices, and at least connect a sata/sas drive?

 

[itronin]

I'm using "inexpensive" pcie bifurcationc SFF-8643 cards and regular SFF-8643 cables. I'm not using special "nvme SFF-8643 to SFF-8643" cables and I think that's been pretty well debunked as hooey. My motherboards are all gen3 so gen3 is all I get and this works great. Use optane 900p's for slog and 905p for data store.

Regarding using different media in the last 4 slots.

while I have not tested it, my understanding is that the data pins do NOT overlap between u.2 & SAS/STA (but could be wrong). If you want to use sata ports from a motherboard then you'll need SATA reverse breakout to SFF-8643 (SM has these). From a SAS controller (93/94/9500) its just SFF-8643 to SFF-8643 to the SAS/SATA connector on the A-N4 that covers those 4 slots. Mine A-N4 has 4 white SFF-8643 (u.2) and 6 black SFF-8643 (sata/sas) - total of 10

I'm about to build an 826 with an N4 backplane. Same difference thing, just fewer ports and they're LFF so I may test it on that build.

no idea on the gen 4 stuff - too rich for my blood.

 

[ano]

for sas, wouldnt that require you to use up x4 connectors? for single drives? ref quad connectors, 1 for each drive

for the reverse breakout ones I'm only finding quad sata to 8643, guessing only p0? needs to be connected

I have some 826 with n4 backplane incoming as well, so we are in the same boat I guess, should have gotten h12ssl-nt (for others reading)

 

[itronin]

for sas, wouldnt that require you to use up x4 connectors? for single drives? ref quad connectors, 1 for each drive

for the reverse breakout ones I'm only finding quad sata to 8643, guessing only p0? needs to be connected

I have some 826 with n4 backplane incoming as well, so we are in the same boat I guess, should have gotten h12ssl-nt (for others reading)

Unless I am misunderstanding your question then no. 1 8643 carries 4 SAS channels - single SFF-8643 cable is 4 drives. If you count the number of black connectors on the 216A-N4 you should find there are 6 and then 4 white ones. 24 sata/sas channels, 4 nvme (each of which is a single SFF-8643)

correct on the reverse breakouts. 4 SATA cables = 1 8643 - remember SAS SATA same cabling, same power. different protocol over the wires. You can also get the reverse breakouts on AMZ. the AMZ cables tend NOT to have SGPIO - I'd look at the bay if you want that.

 

[ericloewe]

while I have not tested it, my understanding is that the data pins do NOT overlap between u.2 & SAS/STA (but could be wrong)

They do not. U.2 works by cramming an extra eight differential pairs into the footprint of the connector, by finding a few surfaces not yet covered with pins (see The Connector Formerly Known as SFF-8639 - Now Called U.2 - PC Perspective ). So, from the backplane point of view, it's great because the backplane is just a bunch of separate traces for SATA/SAS and NVMe, and terrible because it triples the number of traces by adding the eight differential pairs that need to handle at a minimum PCIe 3.0, ideally 4.0 or 5.0.

U.3 cuts this down by sharing pins, but means that whatever is on the other side needs to mux between SATA/SAS and NVMe. Great for Broadcom's sales of tri-mode controllers and expanders, not-so-great for normal people.

U.3 disks are backward compatible with U.2 backplanes.

 

[arnbju]

U.3 disks are backward compatible with U.2 backplanes.

What about U.2 disks in U.3 backplanes?

 

[ericloewe]

Not going to work, unless it's a weird hybrid U.2/U.3 backplane, which is the worst of all worlds:

• The complexity of U.3
• The tricky PCBs of U.2
• Same compatibility as U.2

So, I don't expect anyone to make such a backplane, because it would be pure nonsense.

Fortunately, U.3 is fairly simple to implement on the disk side (because it only has to mux between a few sets of traces, instead of dealing with the SATA/SAS part of the equation), so many/most new designs are likely to be U.3.