Supermicro X11(S|D)Px bifurcation

[Rand__]

So I wanted to run a bifurcated card on my shiny new X11DPI-T and was... befuddled to say the least...

These are the options according to the manual (for each CPU):

CPU1 Configuration/CPU2 Configuration
IOU0 (IIO PCIe Br1)
This item configures the PCI-E Bifuraction setting for a PCI-E port specified by the user.
The options are x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, and Auto

IOU1 (IIO PCIe Br2)
This item configures the PCI-E Bifuraction setting for a PCI-E port specified by the user.
The options are x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, and Auto

IOU2 (IIO PCIe Br3)
This item configures the PCI-E Bifuraction setting for a PCI-E port specified by the user.
The options are x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, and Auto

MCP0 (IIO PCIe Br4)
This item configures the PCI-E Bifuraction setting for a PCI-E port specified by the user.
The options are x16 and Auto

MCP1 (IIO PCIe Br5)
This item configures the PCI-E Bifuraction setting for a PCI-E port specified by the user.
The options are x16 and Auto

So after fiddling for a bit I mailed SM support and they sent me back a nifty little manual how to enable bifurcation. Turns out I was not the first (and will not be the last); so I thought I share it.

Basically you have to look up the CPU to Slot diagram in your boards manual, identify the n-th port per CPU and can then assign that specific port the desired bifurcation level. Things are made more complicated since all PCI-e lanes are listed here and only some are actual slots and only some are bifurcation capable to a certain degree (mostly because they are not bifurcated yet).

 

[bood]

Did this work for you?

I have an X9DRi-F that exposes the same BIOS options, tried toggling them all, matched the block diagram to the slot, nada. It can only see 1 SSD. I don't care about booting from them, I know that would require adding an NVMe module to the UEFI (read: BIOS modding).

 

[Rand__]

Well it did work on the X11 board, not sure how it works on a X9.
But I think there have been discussions re X9 bifurcation in other threads here...

 

[Jorsher]

Thank you. I have an X11SSH-CTF and hoping this guide will help on that too. I want to install an M.2 softraid card.

 

[Rand__]

IIrc bifurcation on the Xeon E3 line was pretty straight forward, so you should be fine

 

[Jorsher]

IIrc bifurcation on the Xeon E3 line was pretty straight forward, so you should be fine

The X11SSH-CTF user manual for the motherboard doesn't mention bifurcation at all... I'll find out for sure when I'm home.

Great board, though. Onboard LSI 3008 and Intel 10gbit. Great for SFF NAS enclosures.

EDIT: Eh, oh well. Was just a want, not a need: https://forums.servethehome.com/ind...on-x11ssh-ctf-with-nvme-m-2-pcie-drive.21686/

 

[Rand__]

Hm...
I'd ask SM support about this before ruling it out.

 

[BennyT]

Hi @Rand__ ,

I also have the x11-DPI. My question for you: Do you know which IOU# for the 1st CPU corresponds with which physical slots:

X11-DPI-N
1st CPU--> slot1:x8 slot2:x16 slot3:x8
2nd CPU--> slot4:x16 slot5:x16 slot6:x16


The X11DPI has a confusing PCIe arrangement (confusing to describe in words). 1stCPU has a physical x16 labeled as slot #2 (and labeled as PCIe #2 on the CPU's end of the diagram), and two x8 physical slots labeled as slot 1 & slot 3. Those two x8 slots 1&3 are already bifurcated/split from a single x16 and labeled as PCIe #1A & #1B on the CPU end of the diagram (but correspond to slots 1 & 3 as mentioned) .

I'm wondering if SLOT 2 (x16 for 1st CPU) can even be bifurcated and if so which IOU# in BIOS corresponds to it in order to make the x4 x4 x4 x4 setting.


I'm purchasing a hyper M.2 NVMe x16 card and trying to foresee which IOU# to set as "x4 x4 x4 x4". Although it should be easy enough to figure it out once i get the card and SSDs and begin experimenting and testing through process of elimination.

Not meaning to complicate it more, but I'm also installing 2nd CPU and that will open up 3 more x16 slots (slots4, 5 and 6). Pretty sure those each correspond to IOU0, IOU2 and IOU3 for 2nd CPU, but won't know for sure until I test it or if you happen to know. Pretty excited to add NVMe storage for my VMs

Thanks

 

[Rand__]

Well it looks like it is slot #2, does it not work? What bifurcation options does it offer?
I am not running mine atm, so can't test myself...

 

[BennyT]

Well it looks like it is slot #2, does it not work? What bifurcation options does it offer?
I am not running mine atm, so can't test myself...

I agree, I think it will be IOU1 for slot#2. Once my card arrives I'll test it. I don't have my PCIe NVMe M.2 expansion card yet but I was trying to plan ahead for when it arrives in mail

In BIOS, for CPU1 bifurcation options:

each IOU# (IOU0, IOU1, IOU2) displays these bifurcation options to choose from
x4x4x4xx4
x4x4x8
x8x8
x16
AUTO

What confused me is if

IOU0 corresponds to slot#1 (an x8 slot) and if
IOU1 corresponds to slot#2 (an x16 slot) and if
IOU2 corresponds to slot#3 (an x8 slot)...

...then some of those bifurcation options obviously won't work since slot1 is only x8 and setting bifurcation to x16 for it wouldn't make sense.

But I think I can test pretty easily once my expansion card arrives. I'll post here again after I get my card and do my testing.

Thanks

 

[Rand__]

I think they are just lazy a**es and use the same config for everything (on multiple boards) ignoring the actual physical attributes...

 

[BennyT]

Successfully installed and configured our PCIE M.2 card into Slot#6 using CPU2 IOU1 set to bifurcation x4x4x4x4. See my attached diagram.
*edit: I set slot#6 to UEFI mode instead of the default setting LEGACY. I think that only is necessary if wanting to passthrough the PCIE card to a guest VM, which I don't plant to do. In other words, I think I could've left it as Legacy.

I may try experiment to move it to slot#2 (I0U1) for CPU1, but since I've already allocated the two m.2 drives to Datastores, I'm unsure if moving the card to a different PCI-E slot will affect that in ESXi.

 

[Rand__]

I may try experiment to move it to slot#2 (I0U1) for CPU1, but since I've already allocated the two m.2 drives to Datastores, I'm unsure if moving the card to a different PCI-E slot will affect that in ESXi.

If you're not passing through it should not matter. Else it would since the PCIe ID will change (i.e. you'd need to re-select for passthrough).
UEFI on Slot can also be needed if you have a more recent Bios on the card and want to use it to boot or to init low level connectivity (HBAs/Raid, NICs)...

 

[Javik50k]

So, x11dph have two m.2 slots, but so far i'm unable to install second SSD. Board simply does not recognize it in second slot. Advice not to use PCIe slot 6 didn't work either. So tried to change both IOU 1 and 2 to 4x4x4x4x and also failed with same result (tried it with only videocard installed, no change). Where should i dig now, please, give some advice!

 

[RolloZ170]

So, x11dph have two m.2 slots, but so far i'm unable to install second SSD

what type of M.2 SSD ?
have you set NVMe Firmware Source to "AMI native" ?

 

[Javik50k]

have you set NVMe Firmware Source to "AMI native" ?

These are Samsung SM961 and 970 Pro. Both are working in first M.2 slot. "AMI Native" is set. Mind using 970 Pro for OS and software, 961 (OEM of 960 Pro) for games etc. BIOS is 2.0, but anyway, i can't use 3.x for ES Xeons.

Also tried to clean M.2 connector, no result.

 

[RolloZ170]

These are Samsung SM961 and 970 Pro. Both are working in first M.2 slot. "AMI Native" is set.

BIOS to defaults ? which BIOS do you use ? can you try recent BIOS to check ?
there is a chance the socket or cpu lacks one of the required pcie lanes (bent pin, bad pcb route, bad cpu lga)

 

[Javik50k]

Firmware Revision : 01.55
BIOS Version: 2.0
BIOS Build Time: 11/30/2017
Redfish Version : 1.0.1
CPLD Version: 02.b6.01

Will try to flash 2.1 tomorrow. Must work with ES (3.x don't have mcodes for ES).

 

[RolloZ170]

Firmware Revision : 01.55
BIOS Version: 2.0
BIOS Build Time: 11/30/2017
Redfish Version : 1.0.1
CPLD Version: 02.b6.01

Will try to flash 2.1 tomorrow. Must work with ES (3.x don't have mcodes for ES).

late but important info, but not too late.
try with Skylake stepping H0 CPU, if it works your ES is the issue.

 

[Javik50k]

Flashed 2.1 today. Still no change, no SSD detected in 2nd M.2. All PCIe connections set to EFI, firmware mode is AMI Native.
Don't know that i am missing now.

Maybe i should just use PCIe to M.2 adapter for 2nd SSD.