NVMe - make the most of your PCI-e Slots (How-to config Supermicro boards for AOC-SLG3-2E4T et al)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

hydra1981

Member
Jul 19, 2015
44
21
8
43
I received an X10SDV in the post just yesterday and I am expecting a X10DRi during the week. I'll see if I can make it work in one of these boards. I'll move the card around in the X10SRi to try different slots in case it makes any difference. I can see in the BIOS that bifurcation is supported only in some slots and not all (just in case it matters)

EDIT: Booting with Ubuntu Live didn't make any difference, can't find the card in lspci.

There is something interesting I noticed. When I first installed the card I powered the machine on and even the mobile rack was dead i.e. no lights no nothing. When I booted the system with a Fedora live image the mobile rack woke up and I can see 1 solid green light next to the disk (i.e. disk found, no activity). Since then it has stayed like that. It is as if something "woke it up" (totally unscientific observation haha)
 
Last edited:

hydra1981

Member
Jul 19, 2015
44
21
8
43
@sth I am playing with the bifurcation settings as I am waiting for a reply from Supermicro support (just in case). I think a read somewhere that it is not very clear which setting in the BIOS corresponds to which PCIe slot i.e. that there is some sort of a label mismatch. Do you have any experience with that? I can see in the BIOS that there is three settings :

IOU2 (IIO1 PCIe Port 1), possible values: x4x4, x8, Auto
IOU0 (IIO1 PCIe Port 2), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto
IOU1 (IIO1 PCIe Port 3), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto

This has me confused. Slots 1, 2 and 3 are x2, x4 and x8 according to the motherboard manual (all x8 physically). Slots 4, 5 and 6 are x8, x4 and x16 with the latter being x16 physically. I think the assumption that the IIO1 PCIe Port 1, 2, 3 settings correspond to slots 1, 2 and 3 respectively might not be right. Why is the BIOS giving me x16 options for ports that do not support me etc. Any idea what is going on there?

(excuse my ignorance but documentation is rather scarce)

Thank you!
 

sth

Active Member
Oct 29, 2015
379
91
28
On the x10sdv it wasn't an issue as there was only one PCI slot. The response from Supermicro was in January 2019 so close to two years ago now, hopefully there been some progress with broadening compatibility since then. Keep us informed.
 

hydra1981

Member
Jul 19, 2015
44
21
8
43
@sth do you reckon that if the card was working fine (regardless of what drives are attached to it) I should be able to see it in Linux with lspci ? Or is it transparent and only the drives themselves turn up in the various "logs"?
 

sth

Active Member
Oct 29, 2015
379
91
28
I think I'm right in thinking this.. although its been a while since I was experimenting so memory could be fuzzy.
It depends on the board you use I think. I have both the simple version which breaks the x16 to 4 x 4x connections, and the one with the more expensive PCI multiplexer hardware onboard which is supposed to be compatible with more systems.
I would expect the bare drives only to show up with the former, with the later, the board itself will likely be shown.

'Simple' version: Supermicro NMVe AOC-SLG3-2E4R
Compat: X10DRL-C, X10DRL-CT, X10DRL-iT, X10DRW-i

'Multiplexer version': Supermicro NVMe AOC-SLG3-2E4
Compat: X10DRU-i+, X10DRW-i, X10DRW-iT, X11SPW-TF, X11SSZ-F, X11SSZ-QF, X11SSZ-TLN4F
 

hydra1981

Member
Jul 19, 2015
44
21
8
43
@sth Thank you. Supermicro just replied and they said the card should work. I am just trying different permutations to first see if the card can be seen. It could be that the card is defective (although I think that is unlikely given that it is new).
 

itronin

Well-Known Member
Nov 24, 2018
1,234
793
113
Denver, Colorado
Did you look in the appropriate motherboard manual at the system block diagram?

Caveat: I have certainly found typos between the manual and what shows up in the BIOS for different boards. When I was figuring out the X9DRD the block diagram was really useful because the cpu/slot pairings were not consecutive nor intuitive to me.

For example: X10DRI page 1-10

Screen Shot 2020-09-21 at 1.52.52 PM.png
 
  • Like
Reactions: Netwerkz101

Netwerkz101

Active Member
Dec 27, 2015
308
90
28
@sth I am playing with the bifurcation settings as I am waiting for a reply from Supermicro support (just in case). I think a read somewhere that it is not very clear which setting in the BIOS corresponds to which PCIe slot i.e. that there is some sort of a label mismatch. Do you have any experience with that? I can see in the BIOS that there is three settings :

IOU2 (IIO1 PCIe Port 1), possible values: x4x4, x8, Auto
IOU0 (IIO1 PCIe Port 2), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto
IOU1 (IIO1 PCIe Port 3), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto

This has me confused. Slots 1, 2 and 3 are x2, x4 and x8 according to the motherboard manual (all x8 physically). Slots 4, 5 and 6 are x8, x4 and x16 with the latter being x16 physically. I think the assumption that the IIO1 PCIe Port 1, 2, 3 settings correspond to slots 1, 2 and 3 respectively might not be right. Why is the BIOS giving me x16 options for ports that do not support me etc. Any idea what is going on there?

(excuse my ignorance but documentation is rather scarce)

Thank you!
It is confusing!! Took some time to get anAsus Hypercard M.2 v1 set up the first time around.
As @itronin pointed out, follow the block diagram for the motherboard.

If your post above (#42) is in relation to the X10SRi, see page 16 of the manual.
Port 1 ---- Slot3 (PCIe x8)
Port 2 ---- Slot6 (PCIe x16)

i would start with those ports/slots set at x4x4 and/or x4x4x4x4 for troubleshooting - leave all other slots empty and set to "auto".
 

hydra1981

Member
Jul 19, 2015
44
21
8
43
Thank you guys, will keep digging. Supermicro said the card should work without tweaking any bifurcation settings (that would make sense for this card!) but the mode needs to be changed to EFI from the default Legacy and that is it. I can see that ESXi (and Linux) can detect the controller, lspci lists 3 entries related to the PLX chipset but the disk is not detected. The SAS cables go to ICY Dock's NVME 2 slot mobile rack


and it could be that the rack is not cooperating. Icy Dock however confirmed that the AOC card has been tested and found working. I am waiting for an Asus Hyper Kit to arrive to see if that would work. In the previous setup I had a £10 no-name PCI Express x8 passive card on the PCI Express slot with the Samsung attached to it and that worked fine. If I stick the Asus on the card and the Samsung in the dock and it works at least we could conclude that all the pieces work they just don't like each other. I'll try the X10SVD, it also has a SAS connection on it (can I use it for testing?)
 

hydra1981

Member
Jul 19, 2015
44
21
8
43

itronin

Well-Known Member
Nov 24, 2018
1,234
793
113
Denver, Colorado
I am also suspecting the cables might be the culprit ...

rather than Supermicro's own CBL-SAST-0623. Could that be the issue? Are these cables so different to each other?
The only thing I've been able to find on what the magic SM cables provide are hot-plug

something to do with a VPP mumble frag mumble... I'm not intune with this to know more.

I wanted to look this up though because I am playing with SM nvme backplanes and a non SM motherboard
 

Eta_Carinae

New Member
Feb 24, 2020
12
2
3
40
Perth, Western Australia
Anyone know whether the AOC-SLG3-4E4T is compatible with the X10DRi-T4+? I note my mb lacks the JNVI1C/JNVI2C headers. Or am I better off with 2 of the AOC-SLG3-2E4? Really don't want to have to upgrade my board & cpus to enable NVMe.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Ive run the 2 port version on x10 boards without cables without issue, so i guess yes. If you have a slot capable of 4x4x4x4x.

Basically any bifurcation capable board should be compatible with the T models (Retimer, active) as they resend the pcie signals opposed to the R models (Redriver,passive) which just retransmit the existing one.
Or so I have been told/understood Supermicro support.
 

Eta_Carinae

New Member
Feb 24, 2020
12
2
3
40
Perth, Western Australia
So, while two NVMe have been working thus far without any serious problems (other than the pcie link problem), I've just installed a 3rd u.2 NVMe, Intel 4610 3.2TB to join the 2x Intel 750 1.2TB and it started of with a blue screen when booting into Windows and now is in a reboot cycle without reaching Windows. :(
 

Attachments

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Hm 4610 is another generation of drives, maybe thats an issue (different nvme version) ? got any other drives to try?
 

Eta_Carinae

New Member
Feb 24, 2020
12
2
3
40
Perth, Western Australia
Hm 4610 is another generation of drives, maybe thats an issue (different nvme version) ? got any other drives to try?
It's definitely the 4610. I thought it was the number of drives, but that didn't affect it. Tried it on its own and no good. Makes the VKVM hang on and off while booting too. These drives are attached to the BPN-SAS3-826EL1-N4 backplane. Maybe it's the backplane, maybe it's the BIOS. Guess I have to take it up with SM.

The X10DRi-T4+ isn't listed as being compatible with NVMe here: Supermicro Solutions Support NVMe SSD in M.2 Form Factor | Supermicro and their QVL HDD/SSD list is down (page was probably throttled by chia miners): Tested Drives List | Supermicro (damn it's not in waybackmachine either).
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Tested drive list (or memory modules or cards or anything basically) does not guarantee its working, its just a non commited information of what has been tested. And even if its today on the list, if its not working tomorrow with a new HW revision its not their problem.

Pretty upset with SM at this point for various reasons, sorry.
 
  • Like
Reactions: Eta_Carinae