Supermicro X9DRi-LN4F+ Ignoring BIOS PCIe Settings - Odd Issue, same on two motherboards

[tbonesteak4dinner]

Hello everyone,
There's a strange issue I'm running into with my used Supermicro X9DRi-LN4F+ board and I was hoping I could get some assistance. There's a lot of info here, and I appreciate anyone with the patience to help:

All PCIe slots are forced to gen 1.1 speeds and are forced to be bifurcated down to x4 level (x16 slot is x4x4x4x4 and x8 slot is x4x4), even if the BIOS is set differently. I have had this issue with two identical motherboards and I'm hoping I'm just missing something.

I can boot into Windows and all 4 of my NVMe drives in my Asus Hyper M.2 card are accessible and running at gen 1.1 speeds, even if the slot is set to x16 and gen 3 in the BIOS. Additionally, the board will not display to a PCIe GPU (tested with 780ti and 1080ti) until after it has booted to Windows. If I disable the onboard video, occasionally it will pick up the addon GPU during initialization, but that is not consistent. I've done my due diligence and as far as I can tell, there is confirmation of this exact board working normally with bifurcated PCIe slots, NVMe drives, and a GPU. Also no, the irony is not lost on me that there is a huge struggle for folks online to find an old SM board that can bifurcate properly while meanwhile, I couldn't get mine to stop if I wanted to.

So far I've tried:
Everything I can think would be relevant in the BIOS. No settings change that all slots are broken into x4 widths and gen 1.1 speed.
TWO different X9DRi-LN4F+ boards, both present with the same issue.
Either the GPU or M.2 card in any x16 slot that will fit them - either alone or both in the system together.
The same CPUs, RAM, addon cards, and PSU with an Intel S2600CO4 motherboard. While that board doesn't bifurcate fully, everything works at full GEN3 speeds.
3.3 and 3.4 BIOS. These are the only two that support bifurcation so I can't test with any others. Both are stock from Supermicro, no mods.
I've also tried diagnosing individual components as best I can.

- CPUs pass Intel Diagnostics, Prime95 causes no crashes, Passmark reports a lower score than expected (by about 10%), but the BIOS revision includes the Spectre fix so this is partially expected. They receive the expected score when in the Intel board which does not include a patch for this.​

- Memory all passes memtest.​

- GPUs both work flawlessly in my other rigs, as does the M.2 card.​

General Specs:
Two Xeon E5-2667 v2 CPUs
512GB Samsung ECC Memory
X9DRi-LN4F+ REV 1.20A
ASUS Hyper M.2 version 1 PCIe card
GTX780ti/1080ti
1000w PSU

I've built and maintained many computers, but I am new to server boards. I would be more than happy for the issue to be with my lack of experience on the platform. The best answer I could hear is "you forgot to check X in the BIOS, ya dork."

 

[MBastian]

Did your try the boards in a case or on a table? Due to the unusual EE-ATX layout make very sure that there aren't any shorts on the back of the board from incompatible standoff screws or ridges in the backplane.
Otherwise, did you update the BIOS to latest and clear the CMOS via JBT1 for good measure?

 

[tbonesteak4dinner]

Did your try the boards in a case or on a table? Due to the unusual EE-ATX layout make very sure that there aren't any shorts on the back of the board from incompatible standoff screws or ridges in the backplane.
Otherwise, did you update the BIOS to latest and clear the CMOS via JBT1 for good measure?

Yes to both, unfortunately no change. I'm using an EEB compatible case so I did have to drill and tap for extra standoffs. Nothing is contacting on the back though. I tested everything just sitting on a box as well.

I've reset the BIOS with the jumper several times on both 3.3 and 3.4 (latest), done recovery flashes and manually cleared the IPMI/BMC settings as well just to be thorough.

This afternoon I swapped both of my CPUs to socket 1 with nothing in socket 2 to 100% rule out the CPUs. No change there either. This is definitely an odd one.

 

[RedX1]

Hello

Are you booting Windows from your NVME drives?

You might want to look at this thread for the Bois mod needed for SM X9 series NVME boot.

https://forums.servethehome.com/ind...ot-with-supermicro-x9da7-x9dri-f.13245/page-2

This worked for me on X9DRI-LN4F v1.20


Good luck.



RedX1

 

[tbonesteak4dinner]

Hello

Are you booting Windows from your NVME drives?

You might want to look at this thread for the Bois mod needed for SM X9 series NVME boot.

https://forums.servethehome.com/ind...ot-with-supermicro-x9da7-x9dri-f.13245/page-2

This worked for me on X9DRI-LN4F v1.20


Good luck.



RedX1

I am not, currently booting from a SATA SSD. Can you tell me a little more about your setup? I assume you're getting full speed on your PCIe lanes (Gen 3). Are you also using a PCIe GPU?

 

[RedX1]

Hello

It is some time since I used this system, as I remember there were no problems with the GPU - when set up to offboard in the Bios.

I did not need to bifurcate, but the system handled 40g Mellanox CX3 Series Nics with no problem at all.

I hope this helps.


RedX1

 

[tbonesteak4dinner]

I was able to get this figured out. While I feel silly, I hate how these types of posts usually vaporize after the OP finds a solution so I wanted to share instead. Apparently I failed to fully complete the recovery flash process. After it flashed, the system booted up normally and showed the new version number in the BIOS. I took this to mean it had completed successfully. All settings were completely functional besides the really strange PCIe behavior. The steps clearly state to boot to DOS and flash again after the recovery flash. One of those RTFM moments, but something I have not experienced on a consumer board before.

After flashing the BIOS again correctly, I now have 4 full speed NVMe drives and fully functional GPU that will get picked up by the system and set as primary VGA. Thanks everyone for jumping in to offer suggestions.

 

[MBastian]

I am still using roughly the same setup as you as my primary workstation. I am glad I've never updated to the latest BIOS revision. I was too lazy to patch NVME boot support into it.
One more note: Do not set the Power Technology to "Max Performance" or your cores won't go over the all-core turbo value See here for details

 

[tbonesteak4dinner]

I am still using roughly the same setup as you as my primary workstation. I am glad I've never updated to the latest BIOS revision. I was too lazy to patch NVME boot support into it.
One more note: Do not set the Power Technology to "Max Performance" or your cores won't go over the all-core turbo value See here for details

Thanks for the heads up! I was using the custom option to manually max everything out but I've been so focused on PCIe that I didn't even check core frequency. I will try that vs the settings you recommend, I'm sure yours will offer an improvement.

 

[RolloZ170]

the motherboard standoffs are mentioned before, but there is another way to short circuit something:

 

[Filez]

I have a different question. How much memory speed (1333 or 800mhz) for a 512GB RAM setup? I have 256GB and need to go to 512gb soon, Thanks.

 

[F1ydave]

I have a different question. How much memory speed (1333 or 800mhz) for a 512GB RAM setup? I have 256GB and need to go to 512gb soon, Thanks.

Did you ever find a solution to this? I am looking at increasing my ram speed since my board is on 1333mhz Samsung right now. Would be nice to go up to 1600/1866.

I found these lots for $118 but not sure if they will work since supermicro's recommended memory page is the worst.

Lot of 8 Samsung 16GB 2Rx4 PC3-14900R Server RAM M393B2G70DB0-CMA (CI) | eBay

Find many great new & used options and get the best deals for Lot of 8 Samsung 16GB 2Rx4 PC3-14900R Server RAM M393B2G70DB0-CMA (CI) at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[Filez]

I got no reply yet

 

[Sean Ho]

Max RAM speed depends on ranks per channel. If you want 512GB on this board, you can get there with 16x 32GB sticks (2DPC), either RDIMM or LRDIMM (both generally 4Rx4). With E5 v2, using RDIMM this would drop you to 800; using LRDIMM (with dual logical ranks) you can stay at 1600 (but not 1866).

As a reminder, you cannot mix RDIMM and LRDIMM.

 

[Filez]

Max RAM speed depends on ranks per channel. If you want 512GB on this board, you can get there with 16x 32GB sticks (2DPC), either RDIMM or LRDIMM (both generally 4Rx4). With E5 v2, using RDIMM this would drop you to 800; using LRDIMM (with dual logical ranks) you can stay at 1600 (but not 1866).

As a reminder, you cannot mix RDIMM and LRDIMM.

I have been using LRDIMM (8x32GB, 1866) on X9DRI-LNF4+ but can not get more than 1333. Is there a setting I am forgetting?

 

[RolloZ170]

I have been using LRDIMM (8x32GB, 1866) on X9DRI-LNF4+ but can not get more than 1333. Is there a setting I am forgetting?

§ Advanced >> Chipset Configuration >> North Bridge >> Memory Configuration >> Enforce POR -> Disabled (default Enabled)
§ Advanced >> Chipset Configuration >> North Bridge >> Memory Configuration >> Frequency >> 1866 (default Auto)

these options can be located at other places in your BIOS.

 

[Filez]

§ Advanced >> Chipset Configuration >> North Bridge >> Memory Configuration >> Enforce POR -> Disabled (default Enabled)
§ Advanced >> Chipset Configuration >> North Bridge >> Memory Configuration >> Frequency >> 1866 (default Auto)

these options can be located at other places in your BIOS.

Thank you thank you. Much appreciated.