Finally: Overclocking EPYC Rome ES

[Zhang]

Furthermore there’s the ‘lock frequency’ checkbox. This can help lift multi-core performance when not using the EDC “bug” on all the CPUs.

@ExecutableFix Thank you for the great overclocking tool!

I am running Linux and I am using the Linux tool by @I.nfraR.ed. Unfortunately there is currently no such option to lock frequency on Linux. @ExecutableFix could you please let me know how this "lock frequency" feature is implemented? Which SMU register it is changing to enable locked frequency? I can add it to the Linux tool.

Thanks!

 

[efschu3]

@ExecutableFix
I would realy appreciate that too. You may share your Windows-Tool sources, so we (Linux guys) can add the missing parts.

 

[ExecutableFix]

I see you were all waiting for the project to become open-source, so here it is: FIFARenderZ/EPYC-Rome-Overclock

If you have any additions feel free to create a merge request

 

[traderjay]

Can you overclock retail Rome chips?

 

[ari2asem]

Can you overclock retail Rome chips?

nope, please read the OP

 

[Layla]

I've finally got my hands on an H11DSi-NT (rev 1) board (after mistakenly receiving a non-NT board first). I have two 2S1404E2VJUG5 (64-core) CPUs.

After updating the BIOS using Rome_H11DSI_Rev1 via IPMI web interface, with a single CPU installed on the left (CPU1) socket (plus RAM), the CPU fan starts to spin, but the board does not POST.

Are there any known issues with trying to use one of these CPUs at a time? Has anyone run into an issue like this? The Supermicro IPMI website doesn't update any of the sensor readings or other CPU info (it reports still having Dual 7551 installed with BIOS 1.3c from the previous owner).

 

[Layla]

I've finally got my hands on an H11DSi-NT (rev 1) board (after mistakenly receiving a non-NT board first). I have two 2S1404E2VJUG5 (64-core) CPUs.

After updating the BIOS using Rome_H11DSI_Rev1 via IPMI web interface, with a single CPU installed on the left (CPU1) socket (plus RAM), the CPU fan starts to spin, but the board does not POST.

Are there any known issues with trying to use one of these CPUs at a time? Has anyone run into an issue like this? The Supermicro IPMI website doesn't update any of the sensor readings or other CPU info (it reports still having Dual 7551 installed with BIOS 1.3c from the previous owner).

Interesting findings:
1. Can't set the date and time (I set it via IPMI Web UI, but it doesn't "stick"). I assume this needs a POST to stick.
2. Sensors all report "Not Present!"
3. IKVM HTML5 just has a grey box.
4. Post Snooping doesn't update - stuck at d1, but this seems to be from last owner.

 

[Layla]

Interesting findings:
1. Can't set the date and time (I set it via IPMI Web UI, but it doesn't "stick"). I assume this needs a POST to stick.
2. Sensors all report "Not Present!"
3. IKVM HTML5 just has a grey box.
4. Post Snooping doesn't update - stuck at d1, but this seems to be from last owner.

Tried resetting CMOS. This reset post code to 0xFF (not updating, though), but nothing else appears to have changed.
The speaker was broken off the board during shipping (yay), so I can't hear audible POST beeps for errors, either.
The CPU does get warm (it has a heatsink+fan on it, and still gets warm).

 

[mirrormax]

I've finally got my hands on an H11DSi-NT (rev 1) board (after mistakenly receiving a non-NT board first). I have two 2S1404E2VJUG5 (64-core) CPUs.

After updating the BIOS using Rome_H11DSI_Rev1 via IPMI web interface, with a single CPU installed on the left (CPU1) socket (plus RAM), the CPU fan starts to spin, but the board does not POST.

Are there any known issues with trying to use one of these CPUs at a time? Has anyone run into an issue like this? The Supermicro IPMI website doesn't update any of the sensor readings or other CPU info (it reports still having Dual 7551 installed with BIOS 1.3c from the previous owner).

Yea it needs both cpus to boot

 

[Layla]

Yea it needs both cpus to boot

Thanks, that did it!

 

[Layla]

Thanks, that did it!

P1_SOCRUN

Upper Critical

1.174 Volts

Is this anything to worry about?

 

[Layla]

P1_SOCRUN

Upper Critical

1.174 Volts

Is this anything to worry about?

Added 40mm Noctua fans on the VRMs and seeing this for now at POST.

	VRMCpu1 Temp	Normal	34 degrees C
	VRMCpu2 Temp	Normal	34 degrees C
	VRMSoc1 Temp	Normal	62 degrees C
	VRMSoc2 Temp	Normal	60 degrees C
	VRMP1ABCD Temp	Normal	36 degrees C
	VRMP1EFGH Temp	Normal	35 degrees C
	VRMP2ABCD Temp	Normal	34 degrees C
	VRMP2EFGH Temp	Normal	33 degrees C
	P1-DIMMA1 Temp	Normal	31 degrees C

 

[Layla]

Anyone have issues installing Windows Server (2019 or 2016) on H11DSi?

Starting a new post for debugging this as to not distract from the point of this parent thread:

https://forums.servethehome.com/index.php?threads/h11dsi-with-dual-epyc-failing-to-boot-windows-server-install-2016-and-2019.29454/

TL;DR; You have to turn off SMT in the BIOS before you can install Windows Server 2016 or Windows Server 2019.

 

[Layla]

28k Cinebench R20 using High Multi-Core Preset with 128T:

Settings:
* SMT disabled (2xZS1406E2VJUG5 --> 128 cores/128 threads)
* Otherwise default BIOS settings (IOMMU Disabled, Above 4G Enabled)
* 16x sticks of DDR4 2133 memory (but one memory channel [P1-DIMMG1] not working)

30965 (~31k) Cinebench R20 using High Multi-Core Preset with 256T:

Settings:
* SMT Enabled (128 cores/256 threads)
* IOMMU Enabled, Above 4G Enabled
* 16x sticks of DDR4 2133 memory (but one memory channel [P1-DIMMG1] not working)

 

[Layla]

28k Cinebench R20 using High Multi-Core Preset.

Settings:
* SMT disabled (2xZS1406E2VJUG5 --> 128 cores/128 threads)
* Otherwise default BIOS settings
* 16x sticks of DDR4 2133 memory (but one memory channel [P1-DIMMG1] not working)

Wanted to to the test again with SMT enabled, but the system hangs during POST with code 0x55 (Memory not installed) after changing back to SMT Enabled. Anybody encountered this?

Note that I've already updated Windows Server 2019 and applied the SP3 driver package from SuperMicro.

 

[Layla]

Wanted to to the test again with SMT enabled, but the system hangs during POST with code 0x55 (Memory not installed) after changing back to SMT Enabled. Anybody encountered this?

Note that I've already updated Windows Server 2019 and applied the SP3 driver package from SuperMicro.

Oh yes that's right. I forgot to mention that there are a couple of bios settings that need to be set on the DSI for the ZS cpu's, it's finicky. So turn on IOMMU and disable Above 4G decoding and it should work

So, I just turned on IOMMU (but did not turn off Above 4G Decoding), and now my system boots again. In-case anyone else runs into this.

 

[Spartus]

Layla,

I have nothing to contribute currently, but I want to say thanks as some of your recent posts were helpful to me!

 

[fobus]

Layla,

Can You please run Corona benchmark test with SMT turned on?
And it will be great if You can tell us frequency on all cores during the test. With multicore preset and with standart settings.

 

[Zhang]

Based on @ExecutableFix's open source Windows tool, I modified @I.nfraR.ed's ZenStates for Rome ES on Linux and added a few missing features. My version comes with these improvements:

1. Read/Write PCI config space directly in Python instead of using setpci. I found that setpci sometimes does not work reliably (probably due to race conditions), and in one occasion it wrote a wrong VID (0x0, =1.55V) that almost killed my CPUs.
2. Dual socket support.
3. Implement some missing features, including frequency lock/unlock and reverting to default voltage/frequency.
4. Add a few preset scripts (e.g., see preset_multicore.sh) that work well on my H11DSi with dual ZS1406E2VJUG5 (untested on other systems, so you may need to adjust them for your use case).

Here is my modified ZenStates on Linux. Note that I only used/tested the CLI on a headless server and I did not use the GUI.

 

[vangoose]

So, I just turned on IOMMU (but did not turn off Above 4G Decoding), and now my system boots again. In-case anyone else runs into this.

Isn't that is mentioned in manual? Turn it off during installation and turn it back on after for win 10.