Supermicro H11DSi + 2x7f52 Boot Problems and Spontaneous Restarts

[RolloZ170]

the bios update zip has some poorly documented scripts with crappy examples (at least to me)

crappy ?

1. Save the BIOS update package to your computer.

2. Extract the files from the UEFI folder of the BIOS package to a USB stick.
   (Note: The USB stick doesn't have to be bootable, but has to be formatted
   with the FAT/FAT32 file system.)

3. Plug the USB stick into a USB port, boot to the Build-In UEFI Shell, and
   type FLASH.nsh BIOSname#.### to start the BIOS update:

     Shell> fs0:
     fs0:\> cd UEFI
     fs0:\UEFI> flash.nsh BIOS_H11DSI-0964_20211228_2.4_STDsp.bin

 

[erock]

crappy ?

1. Save the BIOS update package to your computer.

2. Extract the files from the UEFI folder of the BIOS package to a USB stick.
   (Note: The USB stick doesn't have to be bootable, but has to be formatted
   with the FAT/FAT32 file system.)

3. Plug the USB stick into a USB port, boot to the Build-In UEFI Shell, and
   type FLASH.nsh BIOSname#.### to start the BIOS update:

     Shell> fs0:
     fs0:\> cd UEFI
     fs0:\UEFI> flash.nsh BIOS_H11DSI-0964_20211228_2.4_STDsp.bin

Crappy was a bad word choice. I am sure it would make total sense if I gave it a go. The part that lacks clarity to me is the exact syntax of BIOSname#.### (all caps or sometime lower case?, underscore always between BIOS and name?, when to use dashes?, how are the numbers defined?). Also, is the example specific to your motherboard or a general misc case?

 

[RolloZ170]

The part that lacks clarity to me is the exact syntax of BIOSname#.### (all caps or sometime lower case?

exact name of the file.
in the case of BIOS 2.4
BIOS_H11DSI-0964_20211228_2.4_STDsp.bin

 

[erock]

exact name of the file.
in the case of BIOS 2.4
BIOS_H11DSI-0964_20211228_2.4_STDsp.bin
View attachment 31465

OK, Thank you. This make sense now. I didn’t know if there was a disconnect between the nomenclature in the instructions and the files.

 

[RolloZ170]

if you have access to the BMCWebGUI, there is a BIOS update function.
you need a licence to activate, but it is based on your BMCs MAC
go here

Modular conversion, encoding and encryption online

Web app offering modular conversion, encoding and encryption online. Translations are done in the browser without any server interaction. This is an Open Source project, code licensed MIT.

cryptii.com

enter your MAC on the left, after that you have the key on the right.
you need only the first 6 quad charpairs.
i.e.
857a b7a9 4a4c 103e 3a8c c044

 

[erock]

Here is an update on resolving the spontaneous restart and booting problems with the H11DSi mobo and Epyc 7f52 CPU’s.

After ensuring adequate airflow across the mobo I was able to enable SMT (Simultaneous Multi-threading) in BIOS (see CPU Config) and run at full cpu load (stress-ng test for 1 hr). I also tested a case with all cores utilized and 75% of RAM utilized for 2 hours. To ensure adequate airflow I used the ipmitool to set fan speed to full.

So I think the issue with spontaneous shutdowns was associated with VRM temps that were too high as suggested by alex_stief given that airflow seems to have resolve the issue. My initial tests when the problems occurred were done as bench tests outside of the case and the only fans that were running were on the cpu tower coolers. I think this is a bad idea for server boards designed for lots of airflow across the board. I still have no idea why swapping the CPU’s resolved the issue with POST getting stuck on CPU initialization.

So a key learning is that for these high TDP CPU’s in server mobo’s disable SMT when bench testing and only do a limited amount of bench work without adequate airflow across the board. If you are having issues with booting into the OS just disable SMT and only enable it after airflow is set to max using ipmitool. You may also want to install lower TDP CPU’s when setting up the OS and install the higher TDP CPU after this step is done but I think this is unnecessary if SMT is disabled. Also, I did not have to update BIOS from 2.1 to 2.4 to get this to work.

My final setup with functioning SMT is as follows:

• Case: Dark base pro 900 with top ventilation since CPU sockets are rotated 90 degrees relative to non-server mobos and CPU cooler fans point upwards. I also removed all HD trays and the ODD tray to maximize airflow across the mobo.
• Front fans: 3 140mm 1000rpm silent wings 3
• Back fans: 1 140mm 1000rpm silent winds 3
• 2 Noctua NH-U14S coolers with 1500rpm Noctua fans
• All fans running at full speed

If you need guidance on using the ipmitool use the following link:
https://forums.servethehome.com/index.php?resources/supermicro-x9-x10-x11-fan-speed-control.20/.

 

[alex_stief]

Yeah, the dark base pro 900 is an ok case overall. But airflow really is not its strong suit.
Glad to hear that you made some progress.

 

[erock]

Yeah, the dark base pro 900 is an ok case overall. But airflow really is not its strong suit.
Glad to hear that you made some progress.

Yea, I have seen the benchmarking work from Gamers Nexus showing the so-so thermals for the dark base pro 900. I started to move toward this case after seeing a video from Level 1 Techs showing some Supermicro “workstation” builds with the dark base pro that highlighted the favorable top ventilation.

The case is super flexible and airflow improves by opening (or removing) the front door. Positive aspects of this case for the H11DSi are (1) it is large enough to accommodate the 331mm width and (2) it has open slanted vents on the top back of the case which is helpful for good airflow since CPU sockets are rotated 90 degrees and tower coolers blow upward.

I took off the front door and CPU temps dropped by 5-7 degrees. Removing the HDD cages and adding a third front fan led to another 1-2 degree drop. The third front fan is also necessary to produce positive pressure since the two tower cooler fans are blowing air through the top vents.

Do you have any suggestions for full tower cases with that will accommodate the H11DSi with better airflow than the dark base pro 900 and top ventilation? I don’t really need a lot of storage for my use case and have an upcoming build where I could try a different case and compare.

 

[alex_stief]

For air cooling a dual-socket Epyc system, the Phanteks Enthoo Pro 2 is pretty much ideal.

 

[erock]

For air cooling a dual-socket Epyc system, the Phanteks Enthoo Pro 2 is pretty much ideal.

It looks like I wont have to mod the Phanteks case to get good airflow. Quick question… The mobo manual says maximum mobo width is 330mm whereas the H11Dsi width is described as 331mm on the Supermicro site. Do you think 1mm will matter?

Probably not but want to check if you have given this a go….

Phanteks Innovative Computer Hardware Design

 

[alex_stief]

I have the previous version of that case. One of the other benefits of Phanteks cases: The SSI-EEB boards from Supermicro fit without any modifications. The motherboard tray even has all the right holes for standoffs. Don't worry about 1mm, it will fit.

 

[erock]

I have the previous version of that case. One of the other benefits of Phanteks cases: The SSI-EEB boards from Supermicro fit without any modifications. The motherboard tray even has all the right holes for standoffs. Don't worry about 1mm, it will fit.

I will give this case a try. Thanks for the recommendation.

 

[Sean Ho]

I've also fit SSI-EEB boards in the old Enthoo Pro, missing a few standoffs but no problems. But any large tower case may have issues with VRM cooling as the VRM heatsinks aren't terribly big, and the large case provides lots of space for the air to avoid the VRM heatsinks. If you feel inspired you might experiment with fashioning a shroud to divert some air directly through the VRM heatsinks (whilst keeping sufficient air for the rest of the components, of course).

 

[erock]

I've also fit SSI-EEB boards in the old Enthoo Pro, missing a few standoffs but no problems. But any large tower case may have issues with VRM cooling as the VRM heatsinks aren't terribly big, and the large case provides lots of space for the air to avoid the VRM heatsinks. If you feel inspired you might experiment with fashioning a shroud to divert some air directly through the VRM heatsinks (whilst keeping sufficient air for the rest of the components, of course).

This sounds like an interesting project. I have experimented a bit with a CPU shroud on a Threadripper machine (Thelio Major).

Here are some results I obtained from testing an H11DSi + 2x7f52 machine (258GB RAM) in a Dark Base Pro 900 case with three 1000rpm front fans, one 1000rpm back fan and Noctua NH-U14S coolers with 1500rpm fans blowing upwards:

stress-ng —cpu 30 —vm 30 —vm-bytes 75%
————————————————————————-
Test Duration: 4 hours
CPU: 60C, 63C
VRMCPU: 69C, 69C
VRMSoc: 57C, 67C
VRMHA…: 64C, 80C, 74C, 59C
DIMM’s: 49-65C

stress-ng (max cpu’s)
————————————
Test Duration: 2 hours
CPU: 61C, 61C
VRMCPU: 72C, 72C
VRMSoc: 50C, 54C
VRMHA…: 53C, 66C, 54C, 47C
DIMM’s: 38-49C

Temperature were obtained from ipmitool sensors.
Ambient Temperature was around 71F.

Feel free to comment on wether you think additional cooling is needed.