X12SPL-LN4F, get two AOC-SLG3-2M2 running (bifurcation?!)

[3434s5dg2s]

Hello,

I want get two NVMe addon cards AOC-SLG3-2M2 running in my homelab. This does not work, only one is visible.

System spec:
---------------------------------
- Supermicro X12SPL-LN4F
- ESXi 8.0 Upd3
- Card A: Supermicro AOC-SLG3-2M2 in Slot 7 - only one NVMe disk currently in use and recognized
- Card B: Supermicro AOC-SLG3-2M2 in Slot 2 - two NVMe disks installed - none of them is recognized in ESXi
- latest BIOS & BMC firmware installed


I played around with bifurcation settings for card B but it did not improve anything. Card B has no bifurcation active.
NVMe firmware source is set to "AMI Native" like mentioned in the manual.

- Do I need to adjust any jumpers on the AOC-SLG3-2M2?

That is the structure of the IOUx modules. This table was creat by me based on the block diagram of the mainboard manual in order to understand the bifurcation settings.

Still i have no glue what is wrong.

Thanks for the support.

 

[RolloZ170]

X12SPL is not listed as supported but that must not mean it's not...
note they have address jumpers for SMBus. they should not have both same addr. thought.

 

[vincococka]

Have you tried to set BIOS according to section "3-5" of user manual for AOC-SLG3-2M2 (last page) ?

 

[3434s5dg2s]

Have you tried to set BIOS according to section "3-5" of user manual for AOC-SLG3-2M2 (last page) ?

Yes, "AMI Native" is and Bifurcation 4x4x4x4 as well. Still ist does not recognize the card.

 

[3434s5dg2s]

X12SPL is not listed as supported but that must not mean it's not...
note they have address jumpers for SMBus. they should not have both same addr. thought.

Good hint, I have now set the three jumpers from address 0-0-0 to 1-1-1. Unfortunately no success. None of the two NVMe is recognized

Card is not in CPU Slot 4, Bifurcation x4x4x4x4 is enabled for all four IOUx components but still no success.

 

[vincococka]

I have X12SPL-F with bios 2.2a
I have Delock 89045 2* M.2 <-> PCIe 3.0 x8 (similar component to SuperMicro AOC-SLG3-2M2).

I have Delock 89045 in PCIe 3.0 SLOT 3 (IOU1) IIO PCIe port 2 #1A.
But - 1B is SLOT 4 (PCIe 4.0 version) with c-payne to 2x SFF8639 for another NVMe's.

And all is workin' - see screenshots / photo.

 

[3434s5dg2s]

I have X12SPL-F with bios 2.2a
I have Delock 89045 2* M.2 <-> PCIe 3.0 x8 (similar component to SuperMicro AOC-SLG3-2M2).

I have Delock 89045 in PCIe 3.0 SLOT 3 (IOU1) IIO PCIe port 2 #1A.
But - 1B is SLOT 4 (PCIe 4.0 version) with c-payne to 2x SFF8639 for another NVMe's.

And all is workin' - see screenshots / photo.
View attachment 42903
View attachment 42901
View attachment 42902

I tried the same like you but it fails.

Here are my tests I did:

• New Card in Slot 6 with 2x NVMe SSD, old one remove. No Bifurcation! - FAIL
• New Card removed, old Card in Slot 6 with 2x NVMe SSD. No Bifurcation! - FAIL

• New Card removed, old Card in Slot 7 with 2x NVMe. No Bifurcation! - WIN
• New Card in Slot 7 with 2x NVMe SSD, old Card removed. No Bifurcation! - WIN

• New Card in Slot 6 with 2x NVMe SSD, Old Card in Slot 7 with 1x NVMe SSD. Bifurcation active with IOU0 (x4x4x4x4), different SMBUS address! - FAIL

• New Card in Slot 3 with 2x NVMe SSD, Old Card in Slot 4 with 1x NVMe SSD. Bifurcation active with IOU1 (x4x4x4x4), different SMBUS address! - FAIL

• New Card in Slot 3 with 2x NVMe SSD, Old Card in Slot 7 with 1x NVMe SSD. No Bifurcation, different SMBUS address! - FAIL

I have no glue how to continue...

 

[3434s5dg2s]

This topic drives me crazy. I was in contact mit Supermicro support on AOC-SLG3-2M2.
They could not help me.

Ok I thought now its time to buy a new PCIe NVME card. I bought the Delock 89837. Very similar to the one from @vincococka.
But even that card is not at alle recognized! WTF.. No matter which Bifurcation setting I choose it does not appear.

Was trying the Delock card on all slots 1-2-3-4-6. It does not work.

Slot 7 was finallyworking but then I had to remove AOC-SLG3-2M2. So that is no solution. I need to run both !
Slot 5 is occupied with SAS3008.

What is going wrong here? It seems crazy for me :-(

 

[homeserver78]

But even that card is not at alle recognized! WTF.. No matter which Bifurcation setting I choose it does not appear.

I'm sorry, I don't know what the problem is, but just to ensure there are no misunderstandings:

1. The card itself should be transparent to the PCIe bus. I.e. the card is not supposed to be recognized, only the SSDs should be detected.
2. You must use x4x4(x4x4) bifurcation on the corresponding IOU for the drives to be detected! Without bifurcation only one drive can be detected on each IOU (and only if it happens to be connected to the first four lanes).

In your testing above, what exactly does WIN and FAIL mean? How many drives, and which ones, were detected in each case? When you write "No Bifurcation! - WIN" I assume you didn't test that two drives were actually detected on the card since that would be impossible without x4x4 bifurcation.

 

[3434s5dg2s]

So the first results it posted were belonging to the card AOC-SLG3-2M2. I have returned back this card now and instead bought a Delock 89837.

• New Card removed, old Card in Slot 7 with 2x NVMe. No Bifurcation! - WIN
• New Card in Slot 7 with 2x NVMe SSD, old Card removed. No Bifurcation! - WIN

This means the old card AOC-SLG3-2M2 in Slot 7 with 2x NVMe disks installed works. It works when Bifurcation is set to "Auto"
Second sentence means the new AOC-SLG3-2M2 in Slot 7 with 2x NVMe disks installed works. It works when Bifurcation is set to "Auto"
I wanted to emphasize that both cards are not broken.
No Bifurcation means it was kept in "Auto" setting.

I'm sorry, I don't know what the problem is, but just to ensure there are no misunderstandings:

1. The card itself should be transparent to the PCIe bus. I.e. the card is not supposed to be recognized, only the SSDs should be detected.
2. You must use x4x4(x4x4) bifurcation on the corresponding IOU for the drives to be detected! Without bifurcation only one drive can be detected on each IOU (and only if it happens to be connected to the first four lanes).

In your testing above, what exactly does WIN and FAIL mean? How many drives, and which ones, were detected in each case? When you write "No Bifurcation! - WIN" I assume you didn't test that two drives were actually detected on the card since that would be impossible without x4x4 bifurcation.

Yes I agree to your statement 1). In all my test cases I had NVMe SSDs installed.
Regarding statement 2. Correct that is my understanding as well. But no matter if I installed one or two NVMe SSDs non of them was recognized.

 

[vincococka]

So the first results it posted were belonging to the card AOC-SLG3-2M2. I have returned back this card now and instead bought a Delock 89837.

• New Card removed, old Card in Slot 7 with 2x NVMe. No Bifurcation! - WIN
• New Card in Slot 7 with 2x NVMe SSD, old Card removed. No Bifurcation! - WIN

This means the old card AOC-SLG3-2M2 in Slot 7 with 2x NVMe disks installed works. It works when Bifurcation is set to "Auto"
Second sentence means the new AOC-SLG3-2M2 in Slot 7 with 2x NVMe disks installed works. It works when Bifurcation is set to "Auto"
I wanted to emphasize that both cards are not broken.
No Bifurcation means it was kept in "Auto" setting.




Yes I agree to your statement 1). In all my test cases I had NVMe SSDs installed.
Regarding statement 2. Correct that is my understanding as well. But no matter if I installed one or two NVMe SSDs non of them was recognized.

1, Don't you have PCIe slots set to some specific PCIe revision / speed? Like e.g. PCIe 4.0 or so?
Maybe the NVMe drive can't communicate because some "hardcoded setting".

Please see also: SuperMicro FAQ: PCIe device not found

2, Try following :

1. Enter BIOS and on last page find "Load optimized setup defaults" or something like that
2. Save settings & reboot machine
3. Enter BIOS again to set Bifurcation and see what happens

 

[name stolen]

Reading along and considering if I can help, I just want to emphasize for future reference that No Bifurcation =/ (does not equal) Auto Bifurcation. Please don't reference No Bifurcation! when the setting is Auto.

 

[3434s5dg2s]

So now I managed to get time for this topic again. BIOS reset to optimized defaults was done.
Then reboot and enter BIOS again. The only thing I have changed is to active secure boot plus
enabling Bifurcation to x4x4x4x4 on IOU3 as my Delock card is plugged into CPU Slot 2.
No AMI native setting is active or any fixed link rate on the PCIe port.
But its still not working. ESXi boots but I cannot find a NVMe drive.


Well then I thought lets try CPU slot 4 as this is native PCIe Gen4. Same issue here.
Bifurcation set to x4x4x4x4 on IOU1. I thought lets try other settings for CPU slot 4 like fixed Gen4 link speed setting, re train active.
No success....

 

[homeserver78]

fixed Gen4 link speed

I would certainly try to get it working at a slower speed first, like Gen2. Fixing it at Gen4 seems like it would only complicate the troubleshooting.

 

[3434s5dg2s]

I am still stuck. What I found out now on my side is that I cannot get any PCIe card running besides SLOT5 and SLOT7. For that test I tried my HBA on all other slots and it is not being recognised in ESXi.

So for me it seems that it is not related to any Delock or Supermicro AOC card but a general issue? HAve you ever heard of such an issue?

 

[vincococka]

1, CPU is faulty
2, Motherboard is faulty

Isn't PCIe slot/IOU disabled ?
Have you tried to reset everything via jumper?
Can you run "dcdiag" from Intel to verify CPU is OK ?

Can you reseat your CPU and use proper torque while attaching heatsink ? (please see user manual)
It is really important - maybe you don't have "contact" on pins from the bottom of the socket.

Can you try with different CPU ?

 

[3434s5dg2s]

I really hope this is not a hardware defect. Will check that later if IOU is disabled but I don't think so. Where can I find that setting? Near bifurcation settings?

Bios was already reset to optimized defaults. Which jumper do you mean? Does it a bios reset?
Hm system is running stable since 1,5 years. No I do not have any other CPU. You mean the screw from the cooler is tightened to much?

 

[nexox]

You mean the screw from the cooler is tightened to much?

It's pretty common on the large LGA sockets for people not to torque the screws enough, or not in the correct order, which can lead to sections of the CPU that don't properly contact the pins in the socket. If an entire set of PCIe lanes are dead then that's a possible explanation.

 

[vincococka]

It's pretty common on the large LGA sockets for people not to torque the screws enough, or not in the correct order, which can lead to sections of the CPU that don't properly contact the pins in the socket. If an entire set of PCIe lanes are dead then that's a possible explanation.

Definitely - that was my point too.
I used 8 lbf torque on SuperMicro SNK-P0080AP4 and both servers are working fine since assembly.

 

[3434s5dg2s]

1, CPU is faulty
2, Motherboard is faulty

Isn't PCIe slot/IOU disabled ?
Have you tried to reset everything via jumper?
Can you run "dcdiag" from Intel to verify CPU is OK ?

Can you reseat your CPU and use proper torque while attaching heatsink ? (please see user manual)
It is really important - maybe you don't have "contact" on pins from the bottom of the socket.

Can you try with different CPU ?

- PCIe Slot is not disabled under "PCIe/PCI/PnP Configuration"
- I did tighten the screws nearby the cooler more but with no success