Supermicro X11DPi-NT only one CPU is recognized and only one 32GB RAM stick works

[koval]

Hi,
I recently bought:

• Supermicro X11DPi-NT motherboard
• 2x Intel Xeon Gold 6148 2.40GHz 20-Core CPU SR3B6
• 12x Hynix 32GB PC4-2933Y-R 2Rx4 Registered RAM DDR4 PC4-23400 HMA84GR7JJR4N-WM

My PSU is Seasonic Focus GX650. It’s connected to mobo with two 8-Pin connectors and the 24-Pin connector.

The motherboard is running the latest available BIOS version and IPMI:
Firmware Revision: 01.74.14
Firmware Build Time: 11/30/2023
BIOS Version: 4.4
BIOS Build Time: 08/16/2024
Redfish Version: 1.8.0
CPLD Version: 02.B1.0F

Unfortunately I’ve got 2 major and 1 minor issue with this setup:

#1 - the 2nd CPU is not recognized.
#2 - All but one memory sticks for CPU #1 are marked as OK in BIOS. None of the sticks for CPU #2 show up in BIOS (probably because of the CPU socket issue).
#3 - IPMI doesn’t show any sensor readings.

re. #1
The 2nd CPU doesn’t show up in BIOS, neither in IPMI. I swapped the CPUs around and nothing has changed. Only the CPU in slot 1 works. So, it means that both CPUs are fine, and it suggests that something is going on with the motherboard.
Interestingly, with time both CPU coolers get warm, that would suggest that the CPU in slot #2 is drawing energy.

re. #2
RAM is populated according to the manual for 2 CPUs and 12 DIMMS:
CPU1: P1-DIMMC1/P1-DIMMB1/P1-DIMMA1/P1-DIMMD1/P1-DIMME1/P1-DIMMF1
CPU2: P2-DIMMC1/P2-DIMMB1/P2-DIMMA1/P2-DIMMD1/P2-DIMME1/P2-DIMMF1
I’ve also set the RAM speed in BIOS to 2666Mhz (it Auto mode, it was also set to 2666).

The `Memory Topology` page in BIOS shows only P1-DIMMC1 as OK.
When I populated only that bank P1-DIMMC1, the mobo failed to start with “All channel are disabled due to memory test failures”! Yikes!

The Health Event Log in IPMI is full of following entries:

• DIMM mapped out … - Assertion
• Failing DIMM: DIMM location (Uncorrectable memory component found). (P1-DIMMF1) - Assertion entries

I’m bit lost here now.
Any ideas what’s going on here?

Thank you

 

[nexox]

Have you carefully inspected the pins in socket 2 to make sure none are damaged or bent? If this is your first LGA3647 system have you closely followed the CPU cooler installation process?

 

[koval]

Have you carefully inspected the pins in socket 2 to make sure none are damaged or bent?

I did have a look at it with naked eye, and haven't seen anything wrong. I'll check with zoom on my phone.

this is your first LGA3647 system have you closely followed the CPU cooler installation process?

Yes, that's my first LGA3647 system.
Yes, I followed the CPU cooler installation (Noctua NH-D9 DX-3647 4U). I followed the steps outlined in the manual, and I cross checked that with 1 or 2 videos on youtube.

I also checked the power draw to find out whether the 2nd CPU is actually doing something and I think it doesn't.
When the server if powered off, then BMC draws around 14W.
When I'm in BIOS the CPU fans are almost dead silet but power draw goes up to 158W. (TDP for 6148 is 150W).
Lastly, when Ubuntu 24.04 boots up then the CPU fans crank up to the max but the power draw drops to roughly 110W.

 

[nexox]

Make sure you're torquing each screw down fully, I bottom each one out then give it just a little more, maybe 10 degrees rotation, to make sure it's tight, those Noctua coolers use a smaller wrench and it's very long, so expect it to twist a bit before the screws are torqued. I have heard of memory channels not working from leaving one screw half a turn loose.

 

[RolloZ170]

DIMM mapped out … - Assertion

if a Slot is mapped out you will not get them back until you make a full platform reset (cmos,nvram)

 

[sam55todd]

Reset (as per post above).
Small steps, first ensure CPU1 is fine with one RAM, then add another RAM and so on until all RAM slots on CPU1 populated
after C1subsystem is validated with successful boot (and desirably OS install/tests) - proceed with mount of CPU2.
and yes (another post above) - large count LGAs are sensitive to mounting instructions (clean seat/no folded pins, pressure and pressure balance between different screws {even-landing/touchdown}, correct tightening sequence, etc.)
You can even swap CPUs in first LGA socket to make sure both CPUs are fine.

 

[koval]

Thank you all for your advice and especially to @nexox
After inspecting the pins in the socket (I haven't noticed any issues) I reseated the CPU and tightened up the mounting screws a bit more and it worked like a charm.
Both CPUs are working just fine. Moreover, all memory sticks were recognized straight away! Without resorting to "add-one" method.

Happy days!

 

[koval]

btw. do you know any article or video that would explain what are optimal BIOS settings for similar setup?
I searched for a while and found only this video

and few articles on other supermicro boards:

Supermicro Xeon D SuperServer BIOS 2.3 / IPMI 3.88 released, how to upgrade with ease!

PCs, EVs, home tech, efficiency, and more.

tinkertry.com

Recommended BIOS Settings for Supermicro SuperServer SYS-5028D-TN4T

PCs, EVs, home tech, efficiency, and more.

tinkertry.com

Recommended BIOS Settings/Changes for Supermicro Board

I'm currently setting up a new server and since I have never made a build like this before and I'm new with the Supermicro boards I wonder if someone has some tips or recommendations for changes I should do in the BIOS before proceeding with any installation. The following is my server...

www.truenas.com

 

[RolloZ170]

I searched for a while and found only this video

LOL restore from AC loss "stay off"
means your server doesn't boot after AC loss.
ignore this video.