Supermicro H12DSI-NT6 Troubleshooting

[theregoesplanb]

I'm having trouble getting a system that I'm working on to boot.

Motherboard: SupermicroH12DSI-NT6: H12DSi-NT6 | Motherboards | Super Micro Computer, Inc.
CPU (qty 2): EPYC Rome 7642 48 core
Memory (qty 16): Micron MTA36ASF8G72PZ-3G2E1 64GB (1X64GB) 2RX4 PC4-3200AA DDR4-25600R RDIMM

CPU and memory were sourced from eBay and motherboard was open box from Newegg. Someone clearly returned the motherboard as there was thermal paste slathered over the cpu cover and latch. There also seemed to be some corrosive material on the outside of one of the occulink connectors. I installed the CPUs and heatsink with a torque screwdriver set to 14 in-lbs.

When I initially apply power, there is a small green led behind the front panel connector that blinks a bit more than 20 times. If I press the power switch it lights up solid green. I have a display hooked up to the VGA port, but it doesn't receive a signal. Similarly, I have a keyboard hooked up to USB (have tried multiple ports) that doesn't light up. I do have the dedicated IPMI LAN port hooked up and the activity LED blinks yellow (right side), but link led doesn't light up (left side). However, I don't see the labeled BMC mac show up in my dhcp reservations and scanning for it fails. I've reset the CMOS and these behaviors haven't changed. I also removed all but a single stick of RAM (also tried powering up with no ram) and behavior is consistent.

I'm using Noctua NH-U9-TR4-SP3 coolers and for some reason, the fans do not spin up when I apply power. They do briefly spin when I power off. If I use 3 pin chassis fans on the same header, they work as expected. This may be something I'm able to rectify if I can get into the bios. I suspect the motherboard is bad, and am debating whether or not to return this and order another H12DSI-NT6 or switch to a different manufacturer. There aren't many reviews for this particular motherboard, but the couple that exist aren't glowing: Supermicro Motherboard MBD-H12DSI-NT6-B SoC Socket SP3 EPYC7003/7002 Bulk item - Newegg.com

 

[j_h_o]

Do you have all 3 power connectors connected? What chassis and PSU are you using?

 

[theregoesplanb]

Yeah, all three of those are connected. I'm currently just trying to get it running on a bench, so no chassis (just sitting on standoffs). I'm currently using a Seasonic Titanium Ultra SSR-T1000TR (1000W).

 

[theregoesplanb]

I should also note that when the power button is pressed, besides the solid green LED between JF2 and JUID in that picture above (lower right of JF1), the CPUs are definitely powered. They got pretty toasty (noctua heatsink fans don't turn on) the first time I pressed the power switch and it appeared nothing was happening.

 

[theregoesplanb]

Well, one other data point. I attached an LED to to the Power Fail LED, and it lights up when pressing the power button...

I did remove CPU2 in preparation for return, but now wondering if may be there's some kind of incompatibility with the power supply I'm using. I know the power supply itself works fine.

Edit: Scratch that. It's the front panel LED that lights up solid, not the power fail LED. Late and getting cross-eyed. I did double check and the power fail LED does *not* light up.

 

[theregoesplanb]

Motherboard was bad (lesson learned on "open box" box from Newegg). Ordered another one, and it resolved all of these issues.

 

[RolloZ170]

Motherboard was bad (lesson learned on "open box" box from Newegg)

reason for return "don't like" and the board is sold without any check like usualy done at amazingon ?

 

[bytecode64]

this happened with me twice with H12DSI variants, once all fan spun up with no display , other time no boot at all just blinking led.
i bought one from newegg and one from ebay (6 months apart) and both were new.
Not sure what is going on but never faced any issues with H11 series

 

[primant]

i've had 3x of the h11dsi variant. and all three of them brand new had CPU pin issues.
2x had issues with the CPU pin on socket 2, while 1x had a CPU pin issue on socket 1.
Your issue is resolved now obviously, but something for others to be aware of.
If there are issues with socket #1, the system will power on and you'll get a blank screen, but you'll never get anything.
Supermicro replaced all three boards for me

 

[theregoesplanb]

i've had 3x of the h11dsi variant. and all three of them brand new had CPU pin issues.
2x had issues with the CPU pin on socket 2, while 1x had a CPU pin issue on socket 1.
Your issue is resolved now obviously, but something for others to be aware of.
If there are issues with socket #1, the system will power on and you'll get a blank screen, but you'll never get anything.
Supermicro replaced all three boards for me

Well, as it turns out, this was only the start of my problems. I ordered a brand new H12DSI-NT6 that appeared to be working fine, but turned out only to be the start of a long nightmare. As soon as I attempted to get Linux installed on it, I ran into a long series of issues that are documented here: https://forums.servethehome.com/ind...nux-on-supermicro-epyc-h12.40462/#post-382041

I ended up ordering two new CPUs and a different brand of motherboard (Gigabyte MZ72-HB0) cause I had no idea what the problem was. The CPUs showed up first, and resolved all of the issues I was having trying to get Linux running on the H12DSI-NTC. So both of the original CPUs were bad, which I wrote off to the first H12DSI-NT6 killing them off when it turned on without powering the fans. Everything was working, and I was happy..for about two days. Woke up to machine powered off and wouldn't boot. After extensive troubleshooting, confirmed that the CPU in socket 2 was malfunctioning. Couldn't really explain why it failed after a couple of days. Nothing in any logs indicating issues. Maybe I didn't properly torque the heatsink or I got an unlucky chip? I re-arranged peripherals to only use resources from socket 1 and ordered another CPU. Running Supermicro CPU kill count: 3.

I ran the system for a couple of weeks on a single socket and it was perfectly stable. Gigabyte motherboard came in, and I started making plans to return or re-sell it. When the new CPU came in, I meticulously installed it. I also ordered a new heatsink/fan just in case there was a hot spot or something on the one I was using before. Everything was good...for about 3 days. Once again, I woke up to a powered off machine that wouldn't boot. Once again, I discovered that the CPU in socket #2 was dead. New Supermicro cpu kill count: 4.

So at this point, I'm fairly convinced that the H12DSI-NT6 is killing CPUs in socket #2. I pull it, and switch over the one remaining CPU that was still working into the Gigabyte board and order yet another CPU. It comes in, and immediately has problem detecting memory when all slots are populated. Also has weird streaking/striations/discoloration on pads that probably has something to do with it, so I order yet another CPU. I go ahead and leave in the one that's not detecting all RAM installed until the replacement comes in, and the system is stable for over a week. Replacement CPU comes in, and I swap it in. All RAM is detected and system has been up and running at full load for about 2 weeks now with no more dead CPUs. In total, all of this nonsense has cost me weeks of my time troubleshooting and a good deal of money. I've never had issues with Supermicro boards in the past, but I'm going to have a hard time bringing myself to purchase another one in the future.