NVMe - make the most of your PCI-e Slots (How-to config Supermicro boards for AOC-SLG3-2E4T et al)

[Rand__]

So after a lengthy exchange with the Supermicro Support in Europe (those guys are awesome btw) I have finally managed to get my AOC-SLG3-2E4T running as expected.

I had followed @Patrick 's article re NVMe drive attachment with great interest and have successfully used an AOC-SLG3-2E4 (with integrated PLX chipset) in most of my boards. A couple of weeks ago I obtained another card which had peaked my interest, an AOC-SLG3-2E4T.

This card has 3 significant advantages:

1. It supports all X11 boards (which is actually not true but more of that later)
2. It uses OCuLink connectors which are the newest flair and cables are opposed to U2 - PCIE SFF-8639 ones quite cheap ($15)
3. The card itself is quite cheap as well (at least it was when i was checking initially, around $90, but it seems to be difficult to obtain now). But everything described here should apply to its bigger sibling AOC-SLG3-4E4T as well which is available for €120. But beware that you will need a fully configured x16 slot for this card which is hard to come by with below described limitations.

Of course per drive attachment cost is still significantly higher then a SAS/Sata drive, but it can increase drive density significantly if so desired.

This especially holds true for (SM) Xeon E3 boards which have a limited set of PCI-e slots to begin with (usually 3). I was looking for a way to extend the number of drives I can run on a vSan storage node beyond 2 nvme drives (1 slot reserved for connectx-3 card) and this seemed like a good way. Way cheaper then getting a bunch of Supermicro AOC-SLG3-2E4's which also incur a latency penalty due to the PLX chip.


So the following is the result/requirements for Supermicro Xeon E3 v5/6 based boards:

1. The board seems to explicitly require "Slot 5" as this seems to be the slot that usually supports bifurcation. Not all X11 boards have this particular slot and for those there seems to be no bifurcation support. One board that has it (and also with >x4 connectivity) is the X11SSH-F (and variants). The X11SSL-F and variants also have the slot but its only connected x4 electrically so won't be of any use.
Boards that don't have slots 5 include X11SSZ-F and variants so no bifurcation support on these. As a consequence the "supports all x11 boards" statement on the 2E4T webpage will be removed; not sure what it will be replaced with

2. Bifurcation support needs to be set to x4x4 (for the 2E4T). The 4E4T will not be usable here due to x8 electrical attachment on the X11SSH-F. If you have a board with an x16 slot5 it should work.


3. Slot OPROM needs to be set to EFI

4. Mode selection Jumper (top left on 2E4T can be left alone which is 2x4 opposed to 1x8), would need to be adjusted for 4E4T as described in the manual


5. JNVI2C is not needed

During the discussion there was also a hint on potential auto-bifurcation of certain slots that I still need to follow up... (which might enable this card on boards without Slot5 )




After I had been 'playing' with the E3 boards I also found that the X10SRL-F has gained bifurcation support in its latest Bios version 2.0c . Bifurcation support on X10 E5 boards is called IOUO and seems to be available for more slots then on E3. I have not played around with using various slots that seem to be supported, my tries have been limited to Slot 4 (a x8 in x16 slot).

After a bit more back and forth with support I got the 2E4T to run in the X10SRL-F as well. The following are the settings required for Supermicro X10 (E5-v3/v4) based boards:

1. Set Slot 4 (IOUO (IIO1 PCIe Port 2) to x4x4x4x4 (or x8x4x4)


2. Slot OPROM needs to be set to EFI (Highlighted SR-IOV setting does not matter)

4. Mode selection Jumper to 2x4

5. JNVI2C is not needed



I have not run performance comparisons but I don't expect a significant impact...

Hope that helps someone out

Result in ESX:




If somebody has an AOC-SLG3-2E4R and a compatible board it would be interesting to see whether it is behaving identically to the 2E4T since it should share the same basic functionality (requiring bifurcation support) only with a different connector.


Edit: Another advantage that I forgot to mention:
These cards are 2u compatible - I have found only one 2u compatible PCIe - U2 adapter (except m2 converters) before which was using a custom PCIe extension cable and thus limited in positioning due to short cable length

 

[Kryax]

Thanks for the informative post! Do you know if the AOC-SLG3-2E4T works in a x16 slot wired for x8? The AOC-SLG3-4E4T looks to only work in a x16 slot that is wired for a x16 slot with various x4 configurations when you setup bifurcation but there was no manual for the 2 port version so I couldn't tell what the options are. Reason I ask is I am looking at the X11SPL-F motherboard and wondering if this might be compatible or not.

 

[Rand__]

See this thread ESXi crash with AOC-SLG3-2E4 and Intel DC P3600 Passthrough
by @K D for support in X11 E5 boards

But yes it works in a x8 electrical x16 physical slot, thats how the X11SSH is configured

 

[T_Minus]

Looking forward to testing the 'old card' with this new one in terms of latency / low QD IOPs, any plans for that?

Great write up by the way!

 

[Rand__]

Busy with vsan vs scale io atm and not to sure how to measure latency to be honest, but I can try. Dimly remember a test but will need to search for it.

 

[K D]

@Rand__ Great write up.

@Kyrax the AOC-SLG3-2E4T works out of the box with the X11SPL-F board. There was no need to tweak the bios settings.

 

[AJXCR]

Excellent write up!

So I played around with this quite a bit early on, and even with the bifurcation and oprom correctly setup had no luck at all. The one step I had not taken which you've now brought to light is the jumper.

I suppose this warrants breaking out the various NVMe cards I was never able to get operational again!

-Craig

 

[AJXCR]

With all of the confusion surrounding NVMe for the time being, this should be put into the Guides section @Patrick

 

[gea]

Yes , I am confused as well
I looked at Configure PCI-Express* Lanes for Simultaneous Applications

but it did not help to decide what I should set with a
- AOC-SLG3-4E4T (4 NVMe U.2 OcuLink) in a
- Supermicro Mainboard X11SPH-nCTPF in its 16x pci slot

PCI-e port 1 x 16 or auto in bios and 4x4 on the NVMe HBA?
I will get these parts this week

How many U.2 NVMe x4 can I connect maximally?
With 2 OcuLink onboard and the M.2, I suppose
- OcuLink: 2
- M.2: 1
- 16x PCI-e: 4
- 2 x PCI-8: 4

total: 11
beside the onboard 10G and SAS

 

[Rand__]

I'd set the slot to x4x4x4x4 and the card as well
But for K D it worked out of the box so maybe it does for you too

And nice board btw - what are you going to add in the 6 nvme slots?

 

[gea]

I'd set the slot to x4x4x4x4 and the card as well
But for K D it worked out of the box so maybe it does for you too

And nice board btw - what are you going to add in the 6 nvme slots?

I expect this board to be my next default serverboard to replace my current ones.
At the momemt I have 2 x 900P and expect two more 900P next week to check how it scales.

 

[Rand__]

How many U.2 NVMe x4 can I connect maximally?
total: 11

Missed that part somehow
OCuLink 2
- M.2: 1
- 16x PCI-e: 4
- 2 x PCI-8: 3 - one is a x4 electric in x8 physical , the other is shared with the x16 slot

This also means that both x8/x16 slot will need to support bifurcation - no clue if the new E5 boards can do that

 

[gea]

Yes, 10 NVMe then
According to the manual there are settings for 3 pci-e slots so

1 x pcie 16 (4x4): 4 nvme via AOC-SLG3-4E4T
1 x pcie 8 n.a. because first pci is set to 16x
1 x pcie 8: 2 nvme via AOC-SLG3-2E4T
1 x pcie 4x: 1 nvme via M.2 adapter

2 x nvme via onboard OcuLink
1 x nvme via onboard M.2

=10 NVme
Ideal would be a backplane with 8 x U.2 (OcuLink) + 8 - 16 SAS
Hope Supermicro will offer something

 

[zir_blazer]

Nice to see PCIe in-slot bifurcation gaining traction, that will increase compatibility with the cheaper passive adapters.

Since some time ago, I started to think that Motherboards should stop having M.2/U.2 onboard and instead migrate these to passive PCIe Cards adapters. Supporting multiple ports that shares the same PCIe Lanes complicate Motherboard layout due to the addition of PCIe lane passive switchers (Not the PLX ones, but the TI HD3SS3415 and similar) and dramatically increases the amount of tradeoffs that you have to do in some models, when you could have an all-around base Motherboard and move the tradeoffs to the addon adapters.
For example, if you have a Xeon E3 with 2 PCIe 16x Slots that can do the standard bifurcation to 8x/8x, and can also in-slot bifurcate to 4x/4x/4x/4x (First slot) or 4x/4x (Second slot), you can purchase something like a passive PCIe 16x 4 M.2 adapter. If you plug it into the first slot, you can use 4 M.2, or 2 M.2 in the second slot. Plus the convenience of the heatsink for M.2 drivers. It should be more convenient and cheaper that cards with PLX switches or M.2 slot onboard taking a lot of surface area.

 

[anoother]

Thanks so much for this; picked up a 4e4t recently - the 'All X11' compatibility made me think it had a PLX chip. At least now I know it needs bifurcation.

Now, to find something to use it in...

 

[K D]

Yes, 10 NVMe then
According to the manual there are settings for 3 pci-e slots so

1 x pcie 16 (4x4): 4 nvme via AOC-SLG3-4E4T
1 x pcie 8 n.a. because first pci is set to 16x
1 x pcie 8: 2 nvme via AOC-SLG3-2E4T
1 x pcie 4x: 1 nvme via M.2 adapter

2 x nvme via onboard OcuLink
1 x nvme via onboard M.2

=10 NVme
Ideal would be a backplane with 8 x U.2 (OcuLink) + 8 - 16 SAS
Hope Supermicro will offer something

Unfortunately all SM 24 x 2.5 offerings till date seem to be either 20 SAS + 4 NVMe or 22 NVMe + 2SAS or 24 NVMe.

 

[Rand__]

As a consequence the "supports all x11 boards" statement on the 2E4T webpage will be removed; not sure what it will be replaced with

So it has been removed by now. No replacement so not officially compatible any more... :/

 

[sth]

Sorry for the thread bump, any idea if this would work on a X10SDV-8C-TLN4F xeon-d 1541 motherboard? I need to add a pair of NVMe drives and the AOC-SLG3-4E4R doesn't work with my p3700s or HGST drives. Ideally would like to avoid PLX chips additional latency if possible.

 

[Rand__]

If you have bifurcation support (which you should have iirc) then it should work...

 

[sth]

I would haas thought so too with the AOC-SLG3-4E4R which appears to be very similar, but sadly no luck.
I ordered the more expensive version with the PLX chip which is more compatible but Ill order one of these OCuLink versions up as well in the hope it works. Will report back in the next week or so.