Finally: Overclocking EPYC Rome ES

[Layla]

Has anyone found a bios which works for Rome ES (ZS1406E2VJUG5) on HP DL385 Gen10?

From what I've heard, HP BIOS are non-standard, so it's not so simple as just modifying an existing bios. I've also heard there was a BIOS that worked, but HP removed it from the public site. I'm hoping to hold out and buy a dual socket board with PCI-e 4.0 support once those become available, but right now, these ES are not booting in my HP server.

 

[efschu3]

Does frequency down scaling on idle still work when using OC mode?

 

[c3l3x]

If I want to use a Rome ES (100-000000054-04) on a Gigabyte MZ31-AR0 rev 2, how would that work? I think the original bios (R15_F18) is Agesa 1.0.0.4, but I can't be sure. The next BIOS update after that updated Agesa to 1.0.0.5. Does that mean the Rome CPU will not be supported at all on this rev 2 board and I need to put it in a rev 1 board with a modded BIOS that has Rome support?

I guess another option would be the MZ32-AR0 rev 1. It looks like the first BIOS version would be Agesa 1.0.0.3. Not sure if this board supports BIOS update without CPU installed.

 

[Epyc]

For some fun relevant reading.
Anandtech did a review about the supermicro h11DSI and also concluded the extremely limited options of aftermarket EPYC builds.
real recommendation.

The Supermicro H11DSi Motherboard Mini-Review: The Sole Dual EPYC Solution

www.anandtech.com

 

[Epyc]

In a world where you're CPU pair kicks its GPU pair in the ……… when it comes to FP64 aka double precision Flops
On a sidenote has anyone figured out how to increase the ddr voltages? I am getting nowhere with this 1,2V & better timings

 

[Spartus]

Update: nevermind. I followed this guide then successfully flashed via IPMI
Generate SuperMicro IPMI license - Technicus

-----------------------------------------


Hey everyone. I could use a little help on motherboard flash as it's not going as expected.

H11SSL-i using the OP's (ExecutableFix) custom bios (it currently has 2.0b)

flashing using flash.nsh in uefi shell

it goes over and over:

"Reading flash... XX%"
"check bin is multi tank rom ... fail"
'Error:multi tank roms' rom id are not allowed...
[OK]


(or something similar to this...)

Just keeps reading flash over and over and ends without seemingly doing anything.
Boot into bios and it's still 2.0b. (I'm not sure if custom bios provided is 2.0b with old agesa spliced in, or if it should be going back to 2.0?)

Should I use a different technique? The recovery technique? (USB with SUPER.ROM)

Thanks

 

[Spartus]

So I'm testing a ZS17 (32 core) on H11SSL-i 2.0a modified by ExecutableFix

First, the cinebench scores are awesome. Great.

My actual usage, running memory bandwidth intensive CFD codes, were a little less impressive. It's the whole reason I'm using the platform: the cache and memory bandwidth. I regret to report I was getting about 20% slower on on average than the retail CPU. It seemed like a bandwidth issue. I suspected it might be the infinity fabric and went looking for a way to check it's speed. CPU-Z appears to work.

Summary:
the infinity fabric only runs 1:1 at the minimum ddr4 speed of 1866, otherwise it runs at half the RAM speed, with the exception of DDR4-2133, which neither the RAM or IF reported as expected. I skipped some screenshots, but the missing ones are all 1:2 ratio.

Can people please check their setups and report back? Let's see if its stepping dependent, see if the OEM one's are OK, maybe the board matters...

ExecutableFix, thanks for all your help, you're awesome. I have no idea if this can somehow be tool or bios adjusted, but I do have my fingers crossed !









This next one is retail @ 3200, not the ES:


All the rest are ES: Missing are 2400 MHz (1:2) and 2933 (1:2)

ES@3200


ES@2666


ES@2133 (messed up clocks or readings?)


ES@1866 (finally 1:1)


I do realize the readings could be bad, but the benchmarks match the narrative very well

 

[Epyc]

So I'm testing a ZS17 (32 core) on H11SSL-i 2.0a modified by ExecutableFix

First, the cinebench scores are awesome. Great.

My actual usage, running memory bandwidth intensive CFD codes, were a little less impressive. It's the whole reason I'm using the platform: the cache and memory bandwidth. I regret to report I was getting about 20% slower on on average than the retail CPU. It seemed like a bandwidth issue. I suspected it might be the infinity fabric and went looking for a way to check it's speed. CPU-Z appears to work.

Summary:
the infinity fabric only runs 1:1 at the minimum ddr4 speed of 1866, otherwise it runs at half the RAM speed, with the exception of DDR4-2133, which neither the RAM or IF reported as expected. I skipped some screenshots, but the missing ones are all 1:2 ratio.

Can people please check their setups and report back? Let's see if its stepping dependent, see if the OEM one's are OK, maybe the board matters...

ExecutableFix, thanks for all your help, you're awesome. I have no idea if this can somehow be tool or bios adjusted, but I do have my fingers crossed !



I do realize the readings could be bad, but the benchmarks match the narrative very well

Now that i think about it, what you reported is accurate, I have always seen a NB reading around 700-800 something with 2933mhz ddr.
Dit not realize this because with ram you always divide by two but the NB does another dived .
I have been trying to run the fabric with a fixed 1:1 option from the CBS but this resulted in a totally screwed up system, no option to boot and often not even able to clear the cmos so flashing a new bios is the only recovery from it.
But always tried this at 2933 mhz and 3200. something that now seems way outside the real of possibility's.
Now trying to boot with forced 1:1 on 2133mhz but so far not much luck

Another strange thing i came across is that i got Dimm E1 not functional on both cpu's. Tried everything from switching dimms, countless reseating. But the slot just doesnt work on both cpu's

 

[bayleyw]

Can confirm the fabric issue, I had always assumed there was some kind of divider involved as well.



I'm surprised the hit in CFD codes is only 20%, sparse matrix math is about as bandwidth sensitive as real code gets.

 

[bayleyw]

Further investigation: here are my Sandra inter-core communication results for a ZS1406; could someone with a retail or -04 processor post theirs? I'm not familiar with interpreting Sandra results but the database would seem to indicate the score is indeed off by a factor of two.

 

[Spartus]

Can confirm the fabric issue, I had always assumed there was some kind of divider involved as well.


I'm surprised the hit in CFD codes is only 20%, sparse matrix math is about as bandwidth sensitive as real code gets.

Well I'm running small and fast tests, and the larger of the two is more impacted, I wonder if it gets worse as I do more thorough tests

Also, I'm reporting 20% slower when my CB score is nearly 20% higher. so that's closer to 35-40% spread / impact. It's big enough that it stood out immediately and leaves me a bit disappointed, but I'm hopeful there are some mitigation options. Can you force a non 1:1 ratio? My sample runs 933 MHz, maybe we can do 1600:933. I would have been ecstatic with 1% faster than retail.

I'm really interested to see what an overclocked -04 looks like and if it's IF speeds match retail (they should).

 

[bayleyw]

Well I'm running small and fast tests, and the larger of the two is more impacted, I wonder if it gets worse as I do more thorough tests

Also, I'm reporting 20% slower when my CB score is nearly 20% higher. so that's closer to 35-40% spread / impact. It's big enough that it stood out immediately and leaves me a bit disappointed, but I'm hopeful there are some mitigation options. Can you force a non 1:1 ratio? My sample runs 933 MHz, maybe we can do 1600:933. I would have been ecstatic with 1% faster than retail.

I'm really interested to see what an overclocked -04 looks like and if it's IF speeds match retail (they should).

It sounds like the IF ratios don't really work in the BIOSes that have them unlocked, it might be possible to set the fabric speed through the SMU debug commands but @ExecutableFix would be the best one to answer that.
If you drop your clocks so that the CB scores match, how do your CFD results compare? (I'm mostly interested in gathering the data point for posterity, since there's precious little data on how fabric speeds affect scientific workloads).

 

[Spartus]

I put the retail CPU back in for the next week for some important work, I then have a large benchmark suite I can run that collects dozens of data points. I'll run it both at high clocks and at roughly parity as you suggested. I don't have a spare platform yet to run the ES so I'm chip swapping.

 

[blinkenlights]

In a world where you're CPU pair kicks its GPU pair in the ……… when it comes to FP64 aka double precision FlopsView attachment 14099

That's a great comparison of performance levels (yes, EPYC is silly powerful) but I seem to recall reading that much of the recent GPU accelerated work (AI/DL/NN/MV) is moving towards single-precision variables. In which case, your 2x 1080Ti cards are nearly 5x faster than the 2x EPYC processors

 

[Spartus]

Another strange thing i came across is that i got Dimm E1 not functional on both cpu's. Tried everything from switching dimms, countless reseating. But the slot just doesnt work on both cpu's

Just wanted to comment on this. When I installed the ES my dimm A1 disappeared. Had a little mini-panic. reseating CPU brought it back.

could be board or CPUs, heh would be easier to tell if it wasnt E1 on both sockets.

 

[MrCake117]

I'm using some cheap 2133mhz dimm that can be easily overclocked to 2666mhz , however as soon as I enable the overclocking mode the IF automaticaly runs at 1:2 ratio, if I force 1:1 at 2666mhz it doesn't post.

System hardly post and not stable at 2400mhz oced with 1:1.

 

[bayleyw]

At stock clocks (fresh reboot) I'm also seeing a 1:2 ratio.

 

[Epyc]

Just wanted to comment on this. When I installed the ES my dimm A1 disappeared. Had a little mini-panic. reseating CPU brought it back.

could be board or CPUs, heh would be easier to tell if it wasnt E1 on both sockets.

I found it very strange, thought it might be a es sampsue since it's seems duplicated on both cpu's. But I will try and reseat them and see if it works.

Also I tried to do 1:1 on 2133, also no go.
But now for some reason I can't even read the speed of the nb bridge anymore. Before I could. No matter how I reset or do. It just doesn't give value any more
I have found that any adjustment to the if fabric is really sensitive and causes very nasty lock downs with no boot and difficulty with cmos clear

 

[Epyc]

That's a great comparison of performance levels (yes, EPYC is silly powerful) but I seem to recall reading that much of the recent GPU accelerated work (AI/DL/NN/MV) is moving towards single-precision variables. In which case, your 2x 1080Ti cards are nearly 5x faster than the 2x EPYC processors

Yea, but let's be honest that's on really unprecise fp16, integer 8bit and all kind of jensen huang ops/flops and whatever new metric he can magicly create to say: the more you buy, the more you save!
Real scientific calculations and modeling are still mainly done exclusively in fp64
And of course a v100 kicks the cpu in the nuts and the new a100 kicks them to bloody pulp.
But found it a funny metric

All praise the leather jacket

 

[Spartus]

just curious what the chance of getting ES support on a PCIe 4.0 motherboard. Obviously none of them will ship with old agesa, but could it be spliced in?

Also, My. Epyc, how are you forcing 1:1. I know you are having no luck, but you have an older 2S stepping. I have a ZS and am wondering if it can handle slightly higher IF clocks.

Mostly I'm just eagerly waiting to hear how the -04 clocks its IF. I'll happily trade up if that changes things.