NVMe - make the most of your PCI-e Slots (How-to config Supermicro boards for AOC-SLG3-2E4T et al)

hydra1981

New Member
Jul 19, 2015
11
3
3
39
I received an X10SDV in the post just yesterday and I am expecting a X10DRi during the week. I'll see if I can make it work in one of these boards. I'll move the card around in the X10SRi to try different slots in case it makes any difference. I can see in the BIOS that bifurcation is supported only in some slots and not all (just in case it matters)

EDIT: Booting with Ubuntu Live didn't make any difference, can't find the card in lspci.

There is something interesting I noticed. When I first installed the card I powered the machine on and even the mobile rack was dead i.e. no lights no nothing. When I booted the system with a Fedora live image the mobile rack woke up and I can see 1 solid green light next to the disk (i.e. disk found, no activity). Since then it has stayed like that. It is as if something "woke it up" (totally unscientific observation haha)
 
Last edited:

hydra1981

New Member
Jul 19, 2015
11
3
3
39
@sth I am playing with the bifurcation settings as I am waiting for a reply from Supermicro support (just in case). I think a read somewhere that it is not very clear which setting in the BIOS corresponds to which PCIe slot i.e. that there is some sort of a label mismatch. Do you have any experience with that? I can see in the BIOS that there is three settings :

IOU2 (IIO1 PCIe Port 1), possible values: x4x4, x8, Auto
IOU0 (IIO1 PCIe Port 2), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto
IOU1 (IIO1 PCIe Port 3), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto

This has me confused. Slots 1, 2 and 3 are x2, x4 and x8 according to the motherboard manual (all x8 physically). Slots 4, 5 and 6 are x8, x4 and x16 with the latter being x16 physically. I think the assumption that the IIO1 PCIe Port 1, 2, 3 settings correspond to slots 1, 2 and 3 respectively might not be right. Why is the BIOS giving me x16 options for ports that do not support me etc. Any idea what is going on there?

(excuse my ignorance but documentation is rather scarce)

Thank you!
 

sth

Active Member
Oct 29, 2015
304
50
28
On the x10sdv it wasn't an issue as there was only one PCI slot. The response from Supermicro was in January 2019 so close to two years ago now, hopefully there been some progress with broadening compatibility since then. Keep us informed.
 

hydra1981

New Member
Jul 19, 2015
11
3
3
39
@sth do you reckon that if the card was working fine (regardless of what drives are attached to it) I should be able to see it in Linux with lspci ? Or is it transparent and only the drives themselves turn up in the various "logs"?
 

sth

Active Member
Oct 29, 2015
304
50
28
I think I'm right in thinking this.. although its been a while since I was experimenting so memory could be fuzzy.
It depends on the board you use I think. I have both the simple version which breaks the x16 to 4 x 4x connections, and the one with the more expensive PCI multiplexer hardware onboard which is supposed to be compatible with more systems.
I would expect the bare drives only to show up with the former, with the later, the board itself will likely be shown.

'Simple' version: Supermicro NMVe AOC-SLG3-2E4R
Compat: X10DRL-C, X10DRL-CT, X10DRL-iT, X10DRW-i

'Multiplexer version': Supermicro NVMe AOC-SLG3-2E4
Compat: X10DRU-i+, X10DRW-i, X10DRW-iT, X11SPW-TF, X11SSZ-F, X11SSZ-QF, X11SSZ-TLN4F
 

hydra1981

New Member
Jul 19, 2015
11
3
3
39
@sth Thank you. Supermicro just replied and they said the card should work. I am just trying different permutations to first see if the card can be seen. It could be that the card is defective (although I think that is unlikely given that it is new).
 

itronin

Well-Known Member
Nov 24, 2018
451
281
63
Denver, Colorado
Did you look in the appropriate motherboard manual at the system block diagram?

Caveat: I have certainly found typos between the manual and what shows up in the BIOS for different boards. When I was figuring out the X9DRD the block diagram was really useful because the cpu/slot pairings were not consecutive nor intuitive to me.

For example: X10DRI page 1-10

Screen Shot 2020-09-21 at 1.52.52 PM.png
 
  • Like
Reactions: Netwerkz101

Netwerkz101

Active Member
Dec 27, 2015
298
75
28
@sth I am playing with the bifurcation settings as I am waiting for a reply from Supermicro support (just in case). I think a read somewhere that it is not very clear which setting in the BIOS corresponds to which PCIe slot i.e. that there is some sort of a label mismatch. Do you have any experience with that? I can see in the BIOS that there is three settings :

IOU2 (IIO1 PCIe Port 1), possible values: x4x4, x8, Auto
IOU0 (IIO1 PCIe Port 2), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto
IOU1 (IIO1 PCIe Port 3), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto

This has me confused. Slots 1, 2 and 3 are x2, x4 and x8 according to the motherboard manual (all x8 physically). Slots 4, 5 and 6 are x8, x4 and x16 with the latter being x16 physically. I think the assumption that the IIO1 PCIe Port 1, 2, 3 settings correspond to slots 1, 2 and 3 respectively might not be right. Why is the BIOS giving me x16 options for ports that do not support me etc. Any idea what is going on there?

(excuse my ignorance but documentation is rather scarce)

Thank you!
It is confusing!! Took some time to get anAsus Hypercard M.2 v1 set up the first time around.
As @itronin pointed out, follow the block diagram for the motherboard.

If your post above (#42) is in relation to the X10SRi, see page 16 of the manual.
Port 1 ---- Slot3 (PCIe x8)
Port 2 ---- Slot6 (PCIe x16)

i would start with those ports/slots set at x4x4 and/or x4x4x4x4 for troubleshooting - leave all other slots empty and set to "auto".
 

hydra1981

New Member
Jul 19, 2015
11
3
3
39
Thank you guys, will keep digging. Supermicro said the card should work without tweaking any bifurcation settings (that would make sense for this card!) but the mode needs to be changed to EFI from the default Legacy and that is it. I can see that ESXi (and Linux) can detect the controller, lspci lists 3 entries related to the PLX chipset but the disk is not detected. The SAS cables go to ICY Dock's NVME 2 slot mobile rack


and it could be that the rack is not cooperating. Icy Dock however confirmed that the AOC card has been tested and found working. I am waiting for an Asus Hyper Kit to arrive to see if that would work. In the previous setup I had a £10 no-name PCI Express x8 passive card on the PCI Express slot with the Samsung attached to it and that worked fine. If I stick the Asus on the card and the Samsung in the dock and it works at least we could conclude that all the pieces work they just don't like each other. I'll try the X10SVD, it also has a SAS connection on it (can I use it for testing?)
 

hydra1981

New Member
Jul 19, 2015
11
3
3
39

itronin

Well-Known Member
Nov 24, 2018
451
281
63
Denver, Colorado
I am also suspecting the cables might be the culprit ...

rather than Supermicro's own CBL-SAST-0623. Could that be the issue? Are these cables so different to each other?
The only thing I've been able to find on what the magic SM cables provide are hot-plug

something to do with a VPP mumble frag mumble... I'm not intune with this to know more.

I wanted to look this up though because I am playing with SM nvme backplanes and a non SM motherboard