ES Xeon Discussion

[sam55todd]

Well, I've got LGA4677 MB a year ago and it still sits on a shelf because of most likely QS E3 stepping CPU is unstable (have ES D0 too but it doesn't even detect on SuperMicro), it keeps crashing every 2 hrs.. It might be something else though (have x16 different sticks of RAM, spare PSU) but I've made a peace and ready to wait for another year until ebay starts being flooded with reliable and affordable Sapphire/Emeralds CPUs.

 

[RolloZ170]

Well, I've got LGA4677 MB a year ago and it still sits on a shelf because of most likely QS E3 stepping CPU is unstable (have ES D0 too but it doesn't even detect on SuperMicro), it keeps crashing every 2 hrs..

It looks like your FAN stop accident caused damage after all.
thermal throttling is 800mhz, and we got package power of 75 watts in idle (800mhz cores & uncore)
75 watts is way to much to stay away from damages without any cooling left.

 

[Bert]

It looks like your FAN stop accident caused damage after all.
thermal throttling is 800mhz, and we got package power of 75 watts in idle (800mhz cores & uncore)
75 watts is way to much to stay away from damages without any cooling left.

I have a habit of testing cpus without installing coolers. They usually get hot until I get into boot. I haven't damaged any cpu yet. Is this because these are es cpus.

 

[RolloZ170]

it keeps crashing every 2 hrs.. It might be something else though (have x16 different sticks of RAM, spare PSU)

2x 8 populated but only one CPU ?

 

[RolloZ170]

I have a habit of testing cpus without installing coolers. They usually get hot until I get into boot. I haven't damaged any cpu yet.

how many sapphire rapids ?

Is this because these are es cpus.

QS but prod.units draw same in idle.
75W and more is confirmed and to much, they get damage.

 

[Bert]

how many sapphire rapids ?

QS but prod.units draw same in idle.
75W and more is confirmed and to much, they get damage.

No sapphire rapids. Mostly desktop cpus

 

[sam55todd]

2x 8 populated but only one CPU ?

X13DEI has x16 RAM slots (x8 for each CPU), the reason I've described x16 ram sticks was to show that I have spares to try different combinations.
Currently only one RAM stick is plugged in (and I've changed those several times for different module into right slot as per motherboard manual), therefore it's highly unlikely is memory problem.
So yes, it's higher probability of CPU defect (might be my mistake as you mentioned) and low probability of MB.

 

[Bert]

Thank you @RolloZ170 . I am running 8461v D0 QYFU ES2 now with Asus Sage. I am very pleased with the performance for the price! Thanks again!

edit: Specified chip is ES.

 

[RolloZ170]

Thank you @RolloZ170 . I am running 8461v now with Asus Sage. I am very pleased with the performance. Thanks again!

You should specify that it is a stepping D0 QYFU ES2. Some might think that a real 8461v is running on ASUS SAGE SE.

 

[Civiloid]

75W and more is confirmed and to much, they get damage.

Yeah, and I got mine Dual Xeon 8490H ES E0s and the whole thing consume around 330W idle. That is way too much for me to comfortably start without heatsink. That can get motherboard or CPU or both damaged.

Also, at least as far as I see on mine, they don't report correct temperature (they report that CPU is at about ambient room temperature, which is for sure not correct.

 

[RolloZ170]

Yeah, and I got mine Dual Xeon 8490H ES E0s and the whole thing consume around 330W idle.

IDLE is cores and LLC/Mesh at 800mhz. if you use windows profile high perf. your system can not IDLE.

 

[Civiloid]

IDLE is cores and LLC/Mesh at 800mhz. if you use windows profile high perf. your system can not IDLE.

I'm running Linux on that machine and mostly default BIOS settings. If I disable deep sleep (should be somewhat equivalent of high perf) then without load system consumes around 550W.

 

[RolloZ170]

I'm running Linux on that machine and mostly default BIOS settings.

ok that explains anything. linux never sleeps. and we do not know what devices you have beside the CPUs. so 330Watts can be good.
package power of each CPU should be 60-80Watts

 

[Bert]

This thread is amazing but very hard to find the basic information. How about an spreadsheet to track the knowledge.

Motherboard ES CPU Compatibility List

docs.google.com

@RolloZ170 , please update You have all the knowledge

Everyone, please contribute.

 

[Civiloid]

what devices you have beside the CPUs. so 330Watts can be good.

Gigabyte MS73-HB1, 5 Fans and 6 Mellanox'es in PCIe (they are mostly different, each Mellanox adds 20-30W to power consumption, they never sleep and their power consumption doesn't change based on usage), 2TB Samsung 990 Pro NVMe drive (single as OS). So yeah, 330W is pretty good for it, but single-CPU W5-3435X have about 100W lower power consumption.

Also, FWIW - during startup system spins up to about 600W from the power socket, that is until OS is somewhat booted.

linux never sleeps.

That is not entirely true; it can use lower C states. By default, ASPM and other power save features for peripherals are not enabled, though. However, out of all the devices, the only one supporting ASPM is Samsung NVMe, which won't change its power consumption much.

I'll eventually install Windows there for fun, but much later. I'm too exited to play with my net cards on a system where I can use all of them

 

[Civiloid]

How about an spreadsheet to track the knowledge.

You can enable comments and that should enable suggestion mode, then you will be able to accept or reject suggestions.

When I was building my machine, I created a document where I did a small write-up on what I need. It is slightly specific to Switzerland (as some links for hw lead to swiss retailers), have some ebay links that are probably dead by now (they were accurate as of January this year) and specific to my needs (I want to have a lab that would have a lot of Network cards):

Sapphire Rapids Dual-Socket config

Last Update: 17:50 ZRH Changelog: Added a few notes on alternative PSU options. Expanded list of known-to-be-good part numbers for QYxx CPUs. Lists of what to avoid and get - easier to read Updated notes about memory compatibility and added eBay links for it Added examples for coolers and disks...

docs.google.com

Maybe it would be useful for someone.

 

[DHamov]

My new (temporary) Asus w790E Sage with QYFS, 8 x Micron Ram and Samsung PM9A3 (which gets very hot), uses about 330W during booting. 240W idle in Ubuntu 550W under load (measured at the plug). So because the difference between 240 and 550 is so large, my theory was that it is not only the cpu taking so much idle, but also the mainboard and the rest. When everything is turned off the standby of the asus sage is about 10W.

Also i can confirm that Micron mtc20f2085s1rc56br 5600 rdimm works on the Sage, but strangly enough only with Auto settings and manual frequency selection. It also runns on 6000MT, i was afraid to go higher (no overclocking experience).

 

[Bert]

My new (temporary) Asus w790E Sage with QYFS, 8 x Micron Ram and Samsung PM9A3 (which gets very hot), uses about 330W during booting. 240W idle in Ubuntu 550W under load (measured at the plug). So because the difference between 240 and 550 is so large, my theory was that it is not only the cpu taking so much idle, but also the mainboard and the rest. When everything is turned off the standby of the asus sage is about 10W.

Also i can confirm that Micron mtc20f2085s1rc56br 5600 rdimm works on the Sage, but strangly enough only with Auto settings and manual frequency selection. It also runns on 6000MT, i was afraid to go higher (no overclocking experience).

Are you sure memory speed is actually changing?

 

[DHamov]

Are you sure memory speed is actually changing?

Yes i am quite sure it changes. At first i had the issue that i changed the speed but it was just running at 4800, no matter what i did. It drove me crazy, but later i changed something in the bios somewhere at the top to auto instead of manual, and something else (below in the bios) to set the memory speed. And at the next reboot and it actually runs at 5614 or 6014MHz. i can see the real freq in windows taskmanager, and ubuntu.

Before it was all the time just fixed on 4800 also there, and in windows task manager, but then later it seemed to work.
Strangly enough it shows a different product number in the computer mtc20f2085s1rc56BD1, but the number mtc20f2085s1rc56br is what was on the packaging an on the order.

EDIT for fun i tried 6200, now, but it took extra long to boot, and operating system loading seemed slow.
EDIT it also seem to run that speed on stock voltages. At least i cant see any other voltages than this.
EDIT no idea what the clock 1305Mhz means it sort of annoys me.

 

[Civiloid]

Also i can confirm that Micron mtc20f2085s1rc56br 5600 rdimm works on the Sage, but strangly enough only with Auto settings and manual frequency selection. It also runns on 6000MT, i was afraid to go higher (no overclocking experience).

Try to measure bandwidth with STREAM or something like that and ensure that it is changing.

For memory overclocking it is on one hand simple - you change parameters and run tests afterwards to verify it is stable. It must be memory heavy (linpack, y-cruncher) or specialized (memtest86 or something similar) and runtime should be long (hours). If it is unstable or results significantly drops (ideally you should watch for ECC Correctable error counters, if in stock you don't have any and suddenly you have a lot - then abort test), you go back and change parameters.

You should be careful with voltage and try not to overvolt your memory (it will also heatup more and might require active cooling at 6000 MHz when overclocked). Some chips affected more by voltage, some requires relaxing timings. And then you do multiple iterations of that process to get stable. I bet there are a lot of good videos from overclockers on memory overclocking, RDIMMs are not much different in that sense and MC behavior is somewhat similar to desktop intels (it is more likely to start spewing errors than to fail training on boot, as on AMD it is opposite it usually fails training on boot and system refuses to boot)