ES Xeon Discussion

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

RolloZ170

Well-Known Member
Apr 24, 2016
6,720
2,075
113
I'm running Linux on that machine and mostly default BIOS settings.
ok that explains anything. linux never sleeps. and we do not know what devices you have beside the CPUs. so 330Watts can be good.
package power of each CPU should be 60-80Watts
 

Civiloid

Member
Jan 15, 2024
89
59
18
Switzerland
what devices you have beside the CPUs. so 330Watts can be good.
Gigabyte MS73-HB1, 5 Fans and 6 Mellanox'es in PCIe (they are mostly different, each Mellanox adds 20-30W to power consumption, they never sleep and their power consumption doesn't change based on usage), 2TB Samsung 990 Pro NVMe drive (single as OS). So yeah, 330W is pretty good for it, but single-CPU W5-3435X have about 100W lower power consumption.

Also, FWIW - during startup system spins up to about 600W from the power socket, that is until OS is somewhat booted.

linux never sleeps.
That is not entirely true; it can use lower C states. By default, ASPM and other power save features for peripherals are not enabled, though. However, out of all the devices, the only one supporting ASPM is Samsung NVMe, which won't change its power consumption much.

I'll eventually install Windows there for fun, but much later. I'm too exited to play with my net cards on a system where I can use all of them :)
 

Civiloid

Member
Jan 15, 2024
89
59
18
Switzerland
How about an spreadsheet to track the knowledge.
You can enable comments and that should enable suggestion mode, then you will be able to accept or reject suggestions.

When I was building my machine, I created a document where I did a small write-up on what I need. It is slightly specific to Switzerland (as some links for hw lead to swiss retailers), have some ebay links that are probably dead by now (they were accurate as of January this year) and specific to my needs (I want to have a lab that would have a lot of Network cards):

Maybe it would be useful for someone.
 
  • Like
Reactions: Bert

DHamov

Active Member
Jan 12, 2024
112
27
28
My new (temporary) Asus w790E Sage with QYFS, 8 x Micron Ram and Samsung PM9A3 (which gets very hot), uses about 330W during booting. 240W idle in Ubuntu 550W under load (measured at the plug). So because the difference between 240 and 550 is so large, my theory was that it is not only the cpu taking so much idle, but also the mainboard and the rest. When everything is turned off the standby of the asus sage is about 10W.

Also i can confirm that Micron mtc20f2085s1rc56br 5600 rdimm works on the Sage, but strangly enough only with Auto settings and manual frequency selection. It also runns on 6000MT, i was afraid to go higher (no overclocking experience).
 
Last edited:

Bert

Well-Known Member
Mar 31, 2018
954
478
63
45
My new (temporary) Asus w790E Sage with QYFS, 8 x Micron Ram and Samsung PM9A3 (which gets very hot), uses about 330W during booting. 240W idle in Ubuntu 550W under load (measured at the plug). So because the difference between 240 and 550 is so large, my theory was that it is not only the cpu taking so much idle, but also the mainboard and the rest. When everything is turned off the standby of the asus sage is about 10W.

Also i can confirm that Micron mtc20f2085s1rc56br 5600 rdimm works on the Sage, but strangly enough only with Auto settings and manual frequency selection. It also runns on 6000MT, i was afraid to go higher (no overclocking experience).
Are you sure memory speed is actually changing?
 

DHamov

Active Member
Jan 12, 2024
112
27
28
Are you sure memory speed is actually changing?
Yes i am quite sure it changes. At first i had the issue that i changed the speed but it was just running at 4800, no matter what i did. It drove me crazy, but later i changed something in the bios somewhere at the top to auto instead of manual, and something else (below in the bios) to set the memory speed. And at the next reboot and it actually runs at 5614 or 6014MHz. i can see the real freq in windows taskmanager, and ubuntu.
Memspeed_no_serial_LI.jpg
Before it was all the time just fixed on 4800 also there, and in windows task manager, but then later it seemed to work.
Strangly enough it shows a different product number in the computer mtc20f2085s1rc56BD1, but the number mtc20f2085s1rc56br is what was on the packaging an on the order.

EDIT for fun i tried 6200, now, but it took extra long to boot, and operating system loading seemed slow.
EDIT it also seem to run that speed on stock voltages. At least i cant see any other voltages than this.
EDIT no idea what the clock 1305Mhz means it sort of annoys me.
 

Attachments

Last edited:

Civiloid

Member
Jan 15, 2024
89
59
18
Switzerland
Also i can confirm that Micron mtc20f2085s1rc56br 5600 rdimm works on the Sage, but strangly enough only with Auto settings and manual frequency selection. It also runns on 6000MT, i was afraid to go higher (no overclocking experience).
Try to measure bandwidth with STREAM or something like that and ensure that it is changing.

For memory overclocking it is on one hand simple - you change parameters and run tests afterwards to verify it is stable. It must be memory heavy (linpack, y-cruncher) or specialized (memtest86 or something similar) and runtime should be long (hours). If it is unstable or results significantly drops (ideally you should watch for ECC Correctable error counters, if in stock you don't have any and suddenly you have a lot - then abort test), you go back and change parameters.

You should be careful with voltage and try not to overvolt your memory (it will also heatup more and might require active cooling at 6000 MHz when overclocked). Some chips affected more by voltage, some requires relaxing timings. And then you do multiple iterations of that process to get stable. I bet there are a lot of good videos from overclockers on memory overclocking, RDIMMs are not much different in that sense and MC behavior is somewhat similar to desktop intels (it is more likely to start spewing errors than to fail training on boot, as on AMD it is opposite it usually fails training on boot and system refuses to boot)
 

Bert

Well-Known Member
Mar 31, 2018
954
478
63
45
Yes i am quite sure it changes. At first i had the issue that i changed the speed but it was just running at 4800, no matter what i did. It drove me crazy, but later i changed something in the bios somewhere at the top to auto instead of manual, and something else (below in the bios) to set the memory speed. And at the next reboot and it actually runs at 5614 or 6014MHz. i can see the real freq in windows taskmanager, and ubuntu.
View attachment 35487
Before it was all the time just fixed on 4800 also there, and in windows task manager, but then later it seemed to work.
Strangly enough it shows a different product number in the computer mtc20f2085s1rc56BD1, but the number mtc20f2085s1rc56br is what was on the packaging an on the order.
What else did you change in the BIOS ? I have the same set up, I want to try overclocking memory as well.

You can also increase bclck. If you're lucky you may get another 3%.

Also do you mind sharing your cpuz or intel or any other benchmark you get? I want to see if it is worthwhile to spend extra $250 for 8480± over 8461v.
 
Last edited:

Bert

Well-Known Member
Mar 31, 2018
954
478
63
45
Strangely I don't see any amd es qs cpus in this thread . Are there any nicely priced and compatible threadripper pro (3rd and 4th gen)?
 

DHamov

Active Member
Jan 12, 2024
112
27
28
Try to measure bandwidth with STREAM or something like that and ensure that it is changing.
Currently my windows is not booting any more, and the fix will take some time. So i cant run any of the usual benchmarks now.
When windows worked, i used the memory test in the intel overclocking tool for some hours, it was fine at what was presumed to be 6000MT (it showed up in taskmngr as 6014MHz). I can live without squeezing out the last 5-10% or so. So returned it to 5600 again later.

What else did you change in the BIOS ? I have the same set up, I want to try overclocking memory as well.

You can also increase bclck. If you're lucky you may get another 3%.

Also do you mind sharing your cpuz or intel or any other benchmark you get? I want to see if it is worthwhile to spend extra $250 for 8480± over 8461v.
Bios_Memory_settings.jpgBios_Memory_settings2.jpgInkedBios_Memory_settings3b_LI.jpg
Here my settings. I am not sure but i think it is also on stock voltage! At least in ubuntu sudo dmidecode -t memory shows 1.1v configured. My windows is not booting so i cant do or share any of the usual benchmarks now. But when i did benchmark on my QYFS cinebench was not that great 56K -59K CBR23. Sort of the same as on the w790-ACE i had before, with Fury pro 5600. But many others got much better scores, with similar setup. For me in retrospect the difference between 8461v seems not really worth it. 400USD instead of 150USD and maybe 15% more performance. But it's a matter of priority, and in the past i went for the QYFS. But since memory bandwidth directly scales througput for my use, i decided to go and test for dual cpu and see if it doubles it again. When all arives and works i will also sell my QYFS i think.
 

MillionMiles

Member
Aug 22, 2023
93
39
18
It seems no MCC ES can work on X13 board.
Is there a 1.0 BIOS that can make D0/R0 ES work? Do we need to replace the BIOS chip like X12?
 

Civiloid

Member
Jan 15, 2024
89
59
18
Switzerland
Currently my windows is not booting any more, and the fix will take some time.
Yeah, unstable memory can damage system files and make it unbootable. That is the biggest problem with overclocking RAM - it affects everything.

In Windows I would suggest to use either hwinfo or cpu-z to verify parameters, that usually worked better.

I'm not sure that Cinebench is memory bound, easiest way to check just RAM is to run Aida64 memory benchmark and just look at throughput, or compare something that else that is memory-bound, e.x. y-cruncher memory intensive subtests. In Linux - I would suggest verifying that via STREAM. On Linux you can also use EDAC to check for memory errors:
Code:
for i in /sys/devices/system/edac/mc/mc*/dimm*; do mc=$(grep -o 'mc[0-9]\+' <<< ${i}); dimm=$(grep -o 'dimm[0-9]\+' <<< ${i}); echo "${mc}/${dimm}"; echo -n "ce: "; cat ${i}/dimm_ce_count; echo -n "ue: "; cat ${i}/dimm_ue_count; done
If your overclock is successful, that should all be 0's, no matter what even after you had a successful several hours of something memory intensive (your goal is to use as much RAM as possible)
 

DHamov

Active Member
Jan 12, 2024
112
27
28
Yeah, unstable memory can damage system files and make it unbootable. That is the biggest problem with overclocking RAM - it affects everything.
Probably or even certainly that is true, but i am quite sure my boot issue is because i was making some sort of Frankenstein system by installing some very old windows 7 nvidia driver to resurrect sli like nvlink on this system. (nvidia removed the old drivers that supported that from their search list). So the booting issue was probably my own mistake risk. During install the screen froze and i could not reboot. Need windows safe mode or reinstall. I was thinking of plugging the ssd with OS in another system, to see if same drivers work there, but that will probably screw up other things. So i need to clone.. and it all takes to much time and has no priority now.

I just wanted to report that with QYFS and Asus w790 the Micron 5600 server memory works at 5600, 6000 MT stable as in a few hours of intel xtu memory test, plus a week or so memory intensive loads. Honestly i am not very interested in overclocking just wanted to say that it seems possible. Thanks for your script, i will use it after some long runs. At least for now they show all 0's. I will try to run some linux benchmarks later On Windows i did it a week ago, all was fine. Rolloz170 recommended passmark for memory speed.
 
Last edited:
  • Like
Reactions: Civiloid

DHamov

Active Member
Jan 12, 2024
112
27
28
It seems no MCC ES can work on X13 board.
Is there a 1.0 BIOS that can make D0/R0 ES work? Do we need to replace the BIOS chip like X12?
How is that possible all E0-E.. steppings should work on all 741 boards right? i thought that it was one of the few rules that always applied. Does that mean that 5411N QS would also not work? Is that for super micro only X13? Was your Q0KS also not MCC?
 

RolloZ170

Well-Known Member
Apr 24, 2016
6,720
2,075
113
How is that possible all E0-E.. steppings should work on all 741 boards right? i thought that it was one of the few rules that always applied. Does that mean that 5411N QS would also not work? Is that for super micro only X13? Was your Q0KS also not MCC?
cool down and forget all of this.
MM seems to think all MCC ES are stepping D0.
 
  • Like
Reactions: DHamov

MillionMiles

Member
Aug 22, 2023
93
39
18