SuperMicro H11DSi-NT with AMD Epyc 7742 64 Core CPU

[Rislem]

Hi Everyone.

I have seen similiar posts but not the one I was hoping for to get help

I bought the below hardware

Motherboard: H11DSi-NT | Motherboards | Super Micro Computer, Inc. Rev 2 SuperMicro - H11DSi-NT
CPU: 2x https://www.amd.com/en/products/cpu/amd-epyc-7742 - AMD Epyc 7742
RAM: 4x DDR4 RDIMM 64GB 2Rx4 3200 CL22 | MTA36ASF8G72PZ-3G2B2 | Crucial.com - 4x Micron DDR4 RDIMM 64GB 2Rx4 3200 CL22

To my knowledge nothing is damaged or broken,

but no matter what I do, I cant get the server to turn on, I highly suspect its either the ram or the way its plugged in or to little so the 2 below siutations happens

1) I put both CPU in, and put 2x Dimm on each CPU like the manual recommends putting it in C and D for both, the server turns on, but it doesnt go into bios now everything I read says its points to the memory. No matter what different order I put them same result
2) I unplug 1 cpu and only run with 1, if I plug in the power it wont even turn on, again no matter if I use 1 dimm or put in all 4 nothing happens.

So I was hoping someone had a conclusion is this because it needs a minimum of 8 DIMMs in before the CPU works? I would think it can turn on with only 1 DIMM per CPU but again these are monster CPU so I cant figure it out. I am 99% sure all the parts are compatible with each other so there should be no reason for them to not work. The motherboard all the lights that must turn green and blink and nothing red no beeping nothing, I been working for servers for years and my gut tells me this is either not enough memory modules or something isn't right.

Hoping someone has a suggestion I can try. I don't mind buying more memory but if its not the memory would be a large amount of wasted money.

 

[Alex0220]

Have you tried accessing the IPMI interface? Post snooping should tell you exactly whats wrong.

 

[alex_stief]

You don't need more DIMMs. Your combination of hardware must be able to post with 1 DIMM per CPU. Allegedly, it even posts with only 1 DIMM total and 2 CPUs, but let's not nitpick.
When testing with only 1 CPU installed, make sure it's the right socket. "CPU1", the socket towards rear-I/O.
Before doing anything else, make sure you actually got a rev. 2.x board. There should be some sticker or print on the board.
Then do a full CMOS reset as described in the manual, page 50.
Having the CPUs seated properly to get contact on all pins can be a problem with this socket...
Bur yeah, trying to access the system via IPMI for some post error codes would be a good idea.

 

[Wasmachineman_NL]

New or used CPUs?

 

[Rislem]

@Alex0220 with a little effort I was able to get into IPMI strangely enough if I plug in both CPU's and ram and give it power all the fans instantly turns on but the IPMI says its off. However tis quite late where I am from and the FANs is waking up the entire neigherhood so I will look in the morning but thank you I will check there for sure.

@alex_stief I agree that it should, but as I said to Alex0220 I am inside IPMI and even though everything is on, it says off,but giving it a rest for the night I will do as you said, but yes its a rev 2.0 the sticker says it, there is a update for the bios to 2.3 which I am thinking of running. regarding the CPU sealed I HOPE they are in perfectly cause the person I asked to install it does this for a living, he has built multiple threat ripper machines and the cpus are the size and work exactly like threat rippers so I am confident he got them in correct. will update tomorrow

@Wasmachineman_NL Brand spanking new everything.

 

[RolloZ170]

is your PSU (if any) strong enough ?
which display port are you using ?
recommended is the Onboard 15pin VGA connector.
after AC off the first startup takes a long time for memory training&tests.

 

[RageBone]

You need to enter a license key in order to use post snooping

i'm not sure you are correct, but if you are, that is the most stupid thing SM could have done in my opinion.

 

[Alex0220]

i'm not sure you are correct, but if you are, that is the most stupid thing SM could have done in my opinion.

No, I was wrong. But it is the case on some boards. But not on this EPYC one.

 

[gsrcrxsi]

Where did you buy the CPUs from? Did they come with the carrier tray pre-installed?. Make sure the triangle on the CPU matches the tray and socket. I’ve received them backwards before and it resulted in the system being totally unresponsive (wouldn’t power on) as the CPU wasn’t actually seated in the socket nor touching the pins. Luckily nothing was damaged.

also make sure that all the motherboard standoffs have been removed/correctly placed for the layout on the board. I believe one of the upper ones needs to be removed for the EPYC boards. If you’re shorting the back of the board on a standoff I’m sure it can cause all kinds of weird problems like this. And your issue of being powered on (fans running) when the system is “OFF” sounds like some kind of short or power being applied to the 12V when it shouldn’t be. Check the PSU as well.

and as others mentioned, make sure you try with just 1 stick of memory, check the manual for which slots are recommended for only one stick. Might be a good idea to pull the CPUs and inspect the sockets for bent pins too. Accidents happen.

and yes, first power up takes forever on this board. Literally 5-10mins sometimes lol.

 

[Spartus]

oh god, that first power up still scares me it takes so long, even though I know to expect it.

I would definitely focus on making sure IPMI is working, you don't need CPUs or RAM installed even for IPMI to work. Get the mac address from the silkscreen on the board then go check your dhcp server to find out the lease address.

Once you are in IPMI, from there you can add 1 CPU and stick of ram, boot, and check the iKVM for life after a few minutes. It will also let you verify the bios version etc, but any rev 2 bios should work. be sure to power down the board fully before doing any work; I learned that one the hard way. There is still significant life in the board when off due to IPMI, so pull the power.

 

[Rislem]

So I got into the IPMI and the post snooping shows ff a quick goolge says it means fault hardware.

I removed everything except the montherboard and I still get ff but I assume it needs a CPU and ram plugged in, so I am going to put in 1 cpu with ram and see what happens.

 

[Rislem]

I tried every combonation of ram and CPU

If if plug in 1 cpu + ram and put in power, the fans dont come on and IPMI says ff and the host is in a off state
if I plug in 2 cpu + ram and power in, the fans turn on but IPMI says ff and host is in a off state.

I unplugged everything from the board like the drives and everything but no difference.

At this point I dont know, I should probably take it into the supplier and let them have a look. I dont have extra cpu or ram to test it.

 

[Rislem]

Hi Everyone

I just want to start by thanking everyone, I found the issue, I feel a little asemed because a few of you suggested it and I kept saying to myself "come on how can that be the problem"

So a few of you suggested @gsrcrxsi suggested check the "standoffs" so turns out t he person that put it in didnt use the standsoff that come with the box and he just simply screwed it directly into the box.

I thought as a last resort let me take out the board, check all the stand offs, and use the correct screwes and spacers and boom, she is purring like a kitten...

Thank you all for the assist. PROBLEM SOVED AND A TON OF MONEY SAVED

 

[Spartus]

turns out the person that put it in

so you had standoffs in the wrong spots shorting out the back of the board randomly? And you didn't build this (I mean you could be that person, but you implied not).

can we get the name of this supplier that ships ~20+ thousand dollar servers this way, not even functional on arrival?

 

[Rislem]

@Spartus

I should of said it better, I bought it as parts and my friend that builds gaming pcs for a living assembled it, but yes so the box has these spacers you need to screw in at the right places, he didnt he just screwed it on directly. So it wasnt a suppliers fault. Luckily no damages and its all working

 

[gsrcrxsi]

so you had standoffs in the wrong spots shorting out the back of the board randomly? And you didn't build this (I mean you could be that person, but you implied not).

can we get the name of this supplier that ships ~20+ thousand dollar servers this way, not even functional on arrival?

Sounds like they didn’t use any standoffs at all. Just had the board directly mounted to the motherboard tray. Shorting in several places likely. But you’d think it would be obvious that the I/O plate doesn’t line up.

 

[Rislem]

@gsrcrxsi haha you would think xD I am just a software guy I assumed this guy knew what he was doing but he apologized and lesson learned, I knew this guy from my high school days so I know it wasnt intentional or laziness he simply didnt know.

 

[Leo Atreides]

I dont think you should be ashamed, I had this exact problem. I had even masked unused risers with electrical tape as the case didnt have enough risers and 1/3rd of the board was without support. Turned out that the pins on the motherboard had gone through the electrical masks and it wasnt until i noticed some tiny holes in the masks that I figured it out.

 

[dswartz]

@gsrcrxsi haha you would think xD I am just a software guy I assumed this guy knew what he was doing but he apologized and lesson learned, I knew this guy from my high school days so I know it wasnt intentional or laziness he simply didnt know.

Kind of odd that a guy who builds gaming PCs doesn't know about motherboard standoffs!

 

[gsrcrxsi]

Kind of odd that a guy who builds gaming PCs doesn't know about motherboard standoffs!

yeah. the I/O panel not lining up, and PCIe cards not plugging in all the way should be dead giveaways that something is wrong.