ES Xeon Discussion

[DavidWJohnston]

I'm trying to upgrade my Supermicro X11SPM-TF (BIOS 2.0b) from a Xeon Gold ES 6142-QL28 to a Platinum ES 8260-QQ89 I just bought on eBay.

When I pop in the new CPU, my memory is not detected. ("Memory Training" error). I tried hard setting the speed down to 2400, no go.

Seller says to upgrade BIOS but I am skeptical because my understanding is only the 2.0b supports ES CPUs, and my old 6142 works fine so I am reluctant to attempt an upgrade.

I notice in Supermicro's page for the -F - It says BIOS 3.0a/3.2 is required for 2nd gen Scalable.

Has anybody got experience with this board and CPU combo?

Thanks!

 

[RolloZ170]

because my understanding is only the 2.0b supports ES CPUs

Skylake ES. you have cascade lake now. this gen was not alive the time your BIOS was build.

I notice in Supermicro's page for the -F - It says BIOS 3.0a/3.2 is required for 2nd gen Scalable.

Platinum ES 8260-QQ89 is stepping A0 50655
you need the BIOS with 2nd gen support.

 

[DavidWJohnston]

Thank you, @RolloZ170 your expertise on this is much appreciated.

Do you know if I need a special ES BIOS mod or specific version? If it helps, this is the eBay listing: Intel Xeon Platinum 8260 ES QQ89 24C 2.4GHz 3.1/3.9GHz 35.75MB 165W LGA3647 DDR4 | eBay - 4.5 is the latest available on Supermicro's site.

If I get stuck with neither my new or old CPU working, what would be the best way to recover? Is there a way I can backup my current BIOS before I do anything?

I want to have a bit of a plan before I take an action that might require a lot of time to undo. I have 2 of these servers to upgrade, both identical.

Thanks so much

 

[RolloZ170]

Do you know if I need a special ES BIOS mod or specific version? If it helps, this is the eBay listing: Intel Xeon Platinum 8260 ES QQ89 24C 2.4GHz 3.1/3.9GHz 35.75MB 165W LGA3647 DDR4 | eBay - 4.5 is the latest available on Supermicro's site.

update to 3.8 first. imho the QQ89 should work with BIOS 4.5 but not sure.

 

[RolloZ170]

If I get stuck with neither my new or old CPU working, what would be the best way to recover? Is there a way I can backup my current BIOS before I do anything?

i am not sure but you may have to stay on BIOS 3.x after upgrading to it for security reasons.

I want to have a bit of a plan before I take an action that might require a lot of time to undo. I have 2 of these servers to upgrade, both identical.

QQ89 is working with at least BIOS 3.0c - stepping A0 is used by Production units and QS as well.
you should get rid of that BIOS 2.x because so many bugs/vulnerabilities.

 

[DavidWJohnston]

I got both servers done and everything is working now.

First I upgraded IPMI to 1.74.17 then BIOS to 3.8a. I'm not going to risk 4.5 since everything is fine so far. I did both upgrades through the IPMI web GUI.

A couple of hiccups I had to get through:

- After the IPMI upgrade, the remote console didn't work at first, and showed "undefined" where option is supposed to be. Clearing the browser cookies fixed that.

- One of my servers entered a reboot loop with a black screen with fast blink cursor at the point it should boot. I pulled all the PCIe cards, went into setup, and discovered legacy OPROM was enabled in a few places. I set everything to UEFI, that fixed it.

Thanks @RolloZ170 for the advice, @BlueFox for the amazing Supermicro BIOS indexing page.

Dave

 

[Anonymous411]

Anyone know whee I could get an e0 stepping 8490h? I've been trying to get Q03J or a Q03Z, but I purchased from multiple sellers on ebay and have had multiple orders canceled as they actually only have d0 stepping QYFX. I contacted the rest of the sellers with Q03J and Q03Z listing's and they are also claiming only stock of QYFX. Infact almost all listing's of Q03J and Q03Z have been converted to list QYFX as well in the last 24 hkurs.


Makes me think Q03J and Q03Z were coming from a single source even though there were many sellers.

 

[RolloZ170]

Makes me think Q03J and Q03Z were coming from a single source even though there were many sellers.

that is the most usual case fro ebay seller from china.
try goofish, search for Q03J,Q03Z and ask or use agents.
e.g. Q03Z link

 

[custom90gt]

Well I purchased a QYK8 from ebay (at least that's what the picture shows) and I already have 8x32gb DDR5 4800 from my abandoned Epyc build. Now I am curious what people think I should do motherboard wise. I was thinking either the Gigabyte MS03-CE0 or the Asus W790E-Sage? The MS03-CE0 is actually cheaper, but I know it doesn't support overclocking. That's not a big deal to me, this is going to upgrade my 14900k W680 server.

 

[RolloZ170]

Well I purchased a QYK8 from ebay (at least that's what the picture shows) and I already have 8x32gb DDR5 4800 from my abandoned Epyc build. Now I am curious what people think I should do motherboard wise. I was thinking either the Gigabyte MS03-CE0 or the Asus W790E-Sage?

OC on W790 is limited to memory, QYK8 runs already at max. 4Ghz shortly before fused per core current limit.
And BCLK OC affects PCIe USB/Sata too and gives trouble with new PCIe GPU Cards.
On Gigabyte C741 you can use latest BIOS and you do have all PCIe lanes/outputs available.

 

[custom90gt]

OC on W790 is limited to memory, QYK8 runs already at max. 4Ghz shortly before fused per core current limit.
And BCLK OC affects PCIe USB/Sata too and gives trouble with new PCIe GPU Cards.
On Gigabyte C741 you can use latest BIOS and you do have all PCIe lanes/outputs available.

Awesome, I appreciate the info on the PCIe lanes, that is a bigger reason to go with the Gigabyte anyway. Sadly I've corrupted a few OS installs messing with BCLK overclocking in the past hah. I'm going to pick the Gigabyte. Thanks again for all of your help!

 

[RolloZ170]

Awesome, I appreciate the info on the PCIe lanes, that is a bigger reason to go with the Gigabyte anyway. Sadly I've corrupted a few OS installs messing with BCLK overclocking in the past hah. I'm going to pick the Gigabyte. Thanks again for all of your help!

if i had know Gigabyte is an option for you i would have hint to a look to the Emerald Rapids SP stepping A0 ES like Q2SR,Q2SP,Q30G,Q30H,Q2T7.
all have 4Ghz turbo(half of the core count) but you need a BIOS tweak because without they take over 6 minutes to POST.
e.g. Q2SR 64C 350W crrently only goofish/taobao for ￥2000 (+ ship/tax etc)

 

[custom90gt]

if i had know Gigabyte is an option for you i would have hint to a look to the Emerald Rapids SP stepping A0 ES like Q2SR,Q2SP,Q30G,Q30H,Q2T7.
all have 4Ghz turbo(half of the core count) but you need a BIOS tweak because without they take over 6 minutes to POST.
e.g. Q2SR 64C 350W crrently only goofish/taobao for ￥2000 (+ ship/tax etc)

Ah bummer, I shouldn't have made it seem like I had the W790. But having said that, I don't know if I would know how to order from goofish/taobao hah. Are there any advantages of Emerald Rapids over Sapphire?

 

[RolloZ170]

Are there any advantages of Emerald Rapids over Sapphire?

LOL. you can run (jedec)5600 memory with Q2SR and Q30H. they have only 2 chiplets in XCC configuration, and much much more L3 cache.

 

[custom90gt]

LOL. you can run (jedec)5600 memory with Q2SR and Q30H. they have only 2 chiplets in XCC configuration, and much much more L3 cache.

Humm maybe I'll try to figure out how to purchase from goofish and look for a Q2SR and then resell the QYK8 since it's too late to go back on that purchase.

 

[Civiloid]

without they take over 6 minutes to POST.

I kinda think that stability without the tweak is even more of a problem than POST time. I think I've mentioned before that in that case if it boots, system is basically barely stable.

Are there any advantages of Emerald Rapids over Sapphire?

In addition to all of that, if you get one with accelerators you have better chance using them without stability problems.

 

[custom90gt]

I kinda think that stability without the tweak is even more of a problem than POST time. I think I've mentioned before that in that case if it boots, system is basically barely stable.


In addition to all of that, if you get one with accelerators you have better chance using them without stability problems.

Sorry what do you mean with the first part?

Also I may have to do more research but with accelerators?

 

[Civiloid]

Sorry what do you mean with the first part?

I've tried using my Q2SR with just stock BIOS on Gigabyte motherboard and found out by trying that unless you have a special BIOS made by RolloZ170 even after it boots, OS will be unstable. That is not an issue with Rollo's BIOS.

Also I may have to do more research but with accelerators?

Up to you. TLDR - Xeon Sapphire Rapids and Emerald Rapids might have some extra accelerators built in in some quantity.
DSA - I think that one is always available in some extent. There is a part that always should work - if a PCIe device supports that, data can bypass RAM and be available directly in L3 cache (I personally use that for my experiments with Networking). In SPR/EMR that got improved and you now can theoretically do custom pre-post processing of that data on that accelerator directly.
QAT - that might not be available. HW Encryption/compression accelerator.
IAA - in-memory analytics accelerator. Some sort of pre-processing that databases can use, and there are some opensource ones that uses it.
DLB - Dynamic Load Balancer. Hardware queues that software can use to direct PCIe communcation to one or another. At least that is how I understand it, never used it.

See 5th Gen Intel® Xeon® Scalable Processors – Intel® on Demand...

And basically with all Engineering Sample CPUs - there is always a chance that something is broken and was fixed before QS or Production CPUs. For Sapphire Rapids final stepping is E5, and ES are D0 or E0, they have more chances on something being broken and probably terribly broken. Also as far as I remember, accelerators were not changed in EMR and therefore chances for them to work on EMR ES are higher. Same goes for AMX btw, as that was a new feature in SPR.

So if your workloads uses any of that (or you want to play with that) - there is a chance that EMR A0 would be more usable/stable. Just CPU cores though seems to be stable enough for me on both SPR E0 and EMR A0 (my network tests required quiet a lot of data processing that involved AVX2 and AVX512). Basically keep in mind that ES are not production units and there is always non-0 chance that something won't work. And there are more chances that something new, that was changed or introduced in that generation, would be broken.

 

[Anonymous411]

Would anyone happen to have a link to a Q3W7 8592+ on Taobao or Goofish? I couldn't find one using the Taobao android app. I did find one on Goofish using the Android app, but the app has been updated to require an account to get the link. I'm in the US so anytime I create an account it's automatically flagged for verification before I can do anything with it. I was hoping to use Super Buy to get the processor shipped over to me.

 

[Anonymous411]

I've tried using my Q2SR with just stock BIOS on Gigabyte motherboard and found out by trying that unless you have a special BIOS made by RolloZ170 even after it boots, OS will be unstable. That is not an issue with Rollo's BIOS.

Do you know where the instructions for patching or downloading that special bios are? I was looking through his post history and wasn't sure which BIOS you were referring to. I was thinking about trying this on a Gigabyte MS33-AR0 with a Q3W7