NVMe - make the most of your PCI-e Slots (How-to config Supermicro boards for AOC-SLG3-2E4T et al)

[hydra1981]

I received an X10SDV in the post just yesterday and I am expecting a X10DRi during the week. I'll see if I can make it work in one of these boards. I'll move the card around in the X10SRi to try different slots in case it makes any difference. I can see in the BIOS that bifurcation is supported only in some slots and not all (just in case it matters)

EDIT: Booting with Ubuntu Live didn't make any difference, can't find the card in lspci.

There is something interesting I noticed. When I first installed the card I powered the machine on and even the mobile rack was dead i.e. no lights no nothing. When I booted the system with a Fedora live image the mobile rack woke up and I can see 1 solid green light next to the disk (i.e. disk found, no activity). Since then it has stayed like that. It is as if something "woke it up" (totally unscientific observation haha)

 

[hydra1981]

@sth I am playing with the bifurcation settings as I am waiting for a reply from Supermicro support (just in case). I think a read somewhere that it is not very clear which setting in the BIOS corresponds to which PCIe slot i.e. that there is some sort of a label mismatch. Do you have any experience with that? I can see in the BIOS that there is three settings :

IOU2 (IIO1 PCIe Port 1), possible values: x4x4, x8, Auto
IOU0 (IIO1 PCIe Port 2), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto
IOU1 (IIO1 PCIe Port 3), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto

This has me confused. Slots 1, 2 and 3 are x2, x4 and x8 according to the motherboard manual (all x8 physically). Slots 4, 5 and 6 are x8, x4 and x16 with the latter being x16 physically. I think the assumption that the IIO1 PCIe Port 1, 2, 3 settings correspond to slots 1, 2 and 3 respectively might not be right. Why is the BIOS giving me x16 options for ports that do not support me etc. Any idea what is going on there?

(excuse my ignorance but documentation is rather scarce)

Thank you!

 

[sth]

On the x10sdv it wasn't an issue as there was only one PCI slot. The response from Supermicro was in January 2019 so close to two years ago now, hopefully there been some progress with broadening compatibility since then. Keep us informed.

 

[hydra1981]

@sth do you reckon that if the card was working fine (regardless of what drives are attached to it) I should be able to see it in Linux with lspci ? Or is it transparent and only the drives themselves turn up in the various "logs"?

 

[sth]

I think I'm right in thinking this.. although its been a while since I was experimenting so memory could be fuzzy.
It depends on the board you use I think. I have both the simple version which breaks the x16 to 4 x 4x connections, and the one with the more expensive PCI multiplexer hardware onboard which is supposed to be compatible with more systems.
I would expect the bare drives only to show up with the former, with the later, the board itself will likely be shown.

'Simple' version: Supermicro NMVe AOC-SLG3-2E4R
Compat: X10DRL-C, X10DRL-CT, X10DRL-iT, X10DRW-i

'Multiplexer version': Supermicro NVMe AOC-SLG3-2E4
Compat: X10DRU-i+, X10DRW-i, X10DRW-iT, X11SPW-TF, X11SSZ-F, X11SSZ-QF, X11SSZ-TLN4F

 

[hydra1981]

@sth Thank you. Supermicro just replied and they said the card should work. I am just trying different permutations to first see if the card can be seen. It could be that the card is defective (although I think that is unlikely given that it is new).

 

[itronin]

Did you look in the appropriate motherboard manual at the system block diagram?

Caveat: I have certainly found typos between the manual and what shows up in the BIOS for different boards. When I was figuring out the X9DRD the block diagram was really useful because the cpu/slot pairings were not consecutive nor intuitive to me.

For example: X10DRI page 1-10

 

[Netwerkz101]

@sth I am playing with the bifurcation settings as I am waiting for a reply from Supermicro support (just in case). I think a read somewhere that it is not very clear which setting in the BIOS corresponds to which PCIe slot i.e. that there is some sort of a label mismatch. Do you have any experience with that? I can see in the BIOS that there is three settings :

IOU2 (IIO1 PCIe Port 1), possible values: x4x4, x8, Auto
IOU0 (IIO1 PCIe Port 2), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto
IOU1 (IIO1 PCIe Port 3), possible values: x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, Auto

This has me confused. Slots 1, 2 and 3 are x2, x4 and x8 according to the motherboard manual (all x8 physically). Slots 4, 5 and 6 are x8, x4 and x16 with the latter being x16 physically. I think the assumption that the IIO1 PCIe Port 1, 2, 3 settings correspond to slots 1, 2 and 3 respectively might not be right. Why is the BIOS giving me x16 options for ports that do not support me etc. Any idea what is going on there?

(excuse my ignorance but documentation is rather scarce)

Thank you!

It is confusing!! Took some time to get anAsus Hypercard M.2 v1 set up the first time around.
As @itronin pointed out, follow the block diagram for the motherboard.

If your post above (#42) is in relation to the X10SRi, see page 16 of the manual.
Port 1 ---- Slot3 (PCIe x8)
Port 2 ---- Slot6 (PCIe x16)

i would start with those ports/slots set at x4x4 and/or x4x4x4x4 for troubleshooting - leave all other slots empty and set to "auto".

 

[hydra1981]

Thank you guys, will keep digging. Supermicro said the card should work without tweaking any bifurcation settings (that would make sense for this card!) but the mode needs to be changed to EFI from the default Legacy and that is it. I can see that ESXi (and Linux) can detect the controller, lspci lists 3 entries related to the PLX chipset but the disk is not detected. The SAS cables go to ICY Dock's NVME 2 slot mobile rack

MB834M2K-B_2 Bay M.2 PCIe 3.0/4.0 NVMe SSD Mobile Rack for External 3.5" Drive Bay

www.icydock.com

and it could be that the rack is not cooperating. Icy Dock however confirmed that the AOC card has been tested and found working. I am waiting for an Asus Hyper Kit to arrive to see if that would work. In the previous setup I had a £10 no-name PCI Express x8 passive card on the PCI Express slot with the Samsung attached to it and that worked fine. If I stick the Asus on the card and the Samsung in the dock and it works at least we could conclude that all the pieces work they just don't like each other. I'll try the X10SVD, it also has a SAS connection on it (can I use it for testing?)

 

[hydra1981]

I am also suspecting the cables might be the culprit since I am using these

LSI LSI00405 1.0 meter internal cable SFF8643 to SFF8643 (mini SAS HD to mini SAS HD) - Newegg.com

Buy LSI LSI00405 1.0 meter internal cable SFF8643 to SFF8643 (mini SAS HD to mini SAS HD)--Avago Technologies with fast shipping and top-rated customer service. Once you know, you Newegg!

www.newegg.com

rather than Supermicro's own CBL-SAST-0623. Could that be the issue? Are these cables so different to each other?

 

[itronin]

I am also suspecting the cables might be the culprit ...

rather than Supermicro's own CBL-SAST-0623. Could that be the issue? Are these cables so different to each other?

The only thing I've been able to find on what the magic SM cables provide are hot-plug

something to do with a VPP mumble frag mumble... I'm not intune with this to know more.

I wanted to look this up though because I am playing with SM nvme backplanes and a non SM motherboard

 

[hydra1981]

Hi, just an update, I received the cable CBL-SAST-0623 from Supermicro and now everything is working fine. The problem was indeed the cables. Many thanks !!

 

[Eta_Carinae]

Anyone know whether the AOC-SLG3-4E4T is compatible with the X10DRi-T4+? I note my mb lacks the JNVI1C/JNVI2C headers. Or am I better off with 2 of the AOC-SLG3-2E4? Really don't want to have to upgrade my board & cpus to enable NVMe.

 

[Rand__]

Ive run the 2 port version on x10 boards without cables without issue, so i guess yes. If you have a slot capable of 4x4x4x4x.

Basically any bifurcation capable board should be compatible with the T models (Retimer, active) as they resend the pcie signals opposed to the R models (Redriver,passive) which just retransmit the existing one.
Or so I have been told/understood Supermicro support.

 

[Eta_Carinae]

So this happened with the AOC-SLG3-4E4T & X10DRi-T4+


I was wondering why the benchmarks were so low.

I didn't mess with any of the jumpers on the card and set the bifurcation to auto in bios.

 

[Rand__]

And the PCIE Speed option next to it ? sure its on pcie3?

 

[Eta_Carinae]

So, while two NVMe have been working thus far without any serious problems (other than the pcie link problem), I've just installed a 3rd u.2 NVMe, Intel 4610 3.2TB to join the 2x Intel 750 1.2TB and it started of with a blue screen when booting into Windows and now is in a reboot cycle without reaching Windows.

 

[Rand__]

Hm 4610 is another generation of drives, maybe thats an issue (different nvme version) ? got any other drives to try?

 

[Eta_Carinae]

Hm 4610 is another generation of drives, maybe thats an issue (different nvme version) ? got any other drives to try?

It's definitely the 4610. I thought it was the number of drives, but that didn't affect it. Tried it on its own and no good. Makes the VKVM hang on and off while booting too. These drives are attached to the BPN-SAS3-826EL1-N4 backplane. Maybe it's the backplane, maybe it's the BIOS. Guess I have to take it up with SM.

The X10DRi-T4+ isn't listed as being compatible with NVMe here: Supermicro Solutions Support NVMe SSD in M.2 Form Factor | Supermicro and their QVL HDD/SSD list is down (page was probably throttled by chia miners): Tested Drives List | Supermicro (damn it's not in waybackmachine either).

 

[Rand__]

Tested drive list (or memory modules or cards or anything basically) does not guarantee its working, its just a non commited information of what has been tested. And even if its today on the list, if its not working tomorrow with a new HW revision its not their problem.

Pretty upset with SM at this point for various reasons, sorry.