CPU question about Supermicro NVMe backplane

[The Gecko]

Take a moment and look over this PDF: https://www.supermicro.com/manuals/other/BPN-NVMe3-216N-S4.pdf
It's an NVMe backplane that was designed to connect to 12 OCuLink cables. On page 2-3 (below), there is a description jumpers J26 & J27 to select how many drives are connected to which CPU. This has me thoroughly confused. Why should a backplane need to know which drives are connected to which CPU? My collogues have taken the stance that if this backplane is connected to a single socket mobo, then the most drive we could use on this backplane is 14. Respectfully, I disagree.

We have this backplane connected to a Supermicro H12SSL-i motherboard. The H12SSL-i motherboard features a SP3 socket. These sockets can carry up to 128 PCIe lanes, more than enough to run a 24-slot NVMe backplane. I believe that if I populated all five 16x slots plus all two 8x slots from the single-socket H12SSL-i motherboard with retimers and OCuLink cables, I should be able to use all 24 slots of this backplane with NVMe drives, but I am not in a position to test it.

If you own one of these backplanes, I want to hear from you.

 

[The Gecko]

FYI: Motherboard Manual

 

[i386]

It's an NVMe backplane that was designed to connect to 12 OCuLink cables.

I count 24 conenctors...

(Source the linked pdf page 1)

Why should a backplane need to know which drives are connected to which CPU?

Mangement/LED controls

I believe that if I populated all five 16x slots plus all two 8x slots from the single-socket H12SSL-i motherboard with retimers and OCuLink cables, I should be able to use all 24 slots of this backplane with NVMe drives, but I am not in a position to test it.

That's a lot of local bandwidth, no more pcie lanes for fast networking

 

[mattventura]

Sadly the public information on this kind of stuff is very lacking, so most of this comes from inference.

From what I gather, I don't think you'd be able to get management working correctly with that setup anyway. Connecting that many drives would require the use of multiple redriver/retimer cards, but you only get one JNVI2C connector on the board. Thus, while the drives themselves would work, it's unlikely you'd be able to get proper management (and maybe surprise hotplug support?) of more than four drives (possibly +2 from the onboard 8i connector).

Supermicro will most likely tell you that that MB + BP combo isn't supported. I got the same answer when asking about how to configure a bpn-sas3-826el1-n4 for an H12DSI.

I'd also ask what features you're really after? LED control to identify drives? Surprise hotplug?

 

[ano]

I run H12 on older -n4 btw, works very well, you get gen4 speeds

imho this should work, nvme likes to know what is where

hotplug/hotswap on nvme is dodgy at best, no matter the hw/generation though it seeems, but its getting better

 

[oneplane]

That's a lot of local bandwidth, no more pcie lanes for fast networking

Who needs fast networking if you already hoard store the entire internet on your local disks

 

[mattventura]

I run H12 on older -n4 btw, works very well, you get gen4 speeds

But does LED functionality + surprise hotplug work? That's usually the tricky part.

hotplug/hotswap on nvme is dodgy at best, no matter the hw/generation though it seeems, but its getting better

I've found that NVMe switches work the best if you don't care about being on the newest gen. I have surprise hotplug and removal working fine on X9 hardware despite it having no native NVMe support. LEDs don't work, but I can still deduce which drive I need to remove because powering down the PCIe device will turn off the act/presence LED.

 

[i386]

I've found that NVMe switches work the best if you don't care about being on the newest gen.

Is that what you mean with nvme switches: https://docs.broadcom.com/doc/BC00-0443EN ?
Personally I would use retimer add on cards or something with pcie switches

 

[mattventura]

Is that what you mean with nvme switches: https://docs.broadcom.com/doc/BC00-0443EN ?
Personally I would use retimer add on cards or something with pcie switches

PCIe switch cards intended for NVMe. The AOC-SLG3-2E4 is pretty inexpensive and works well.

The main difference between using a retimer/redriver vs a switch is that while both would require some degree of PCIe hotplugging support in the BIOS (AFAIK, it needs to understand hotplug events and understand that it needs to reserve bus numbers for downstream switch ports), the former requires hotplug support on the specific slot(s), while a switch's downstream ports are slots themselves and they can bring their own slot hotplug support.

I will add that while Broadcom's offerings (both NVMe-only and tri-mode) are rather expensive, and have several disadvantages, they are great at giving you hotplug support even on host systems with poor support.

 

[ano]

broadcom tri-mode performance is quite horrible btw. we have run labs on them, your better off with SAS drives... by far!