Where are the DDR5 ECC UDIMMs?

[pd4ever]

With that CPU and that many hard drives, what is the approximate energy consumption you think? Not judging just asking.

Each drive uses 5-10 watts and about 25 watts at spin up. So anywhere from 100-500 watts. The CPU at idle is maybe 50 watts with 250 peak? I got a 1300 watt PSU to be sure I wouldn't be cutting it close. I'll have to see if my HBA allows staggered spin up, but I was planning for worst case.

I'm going to be running some VMs also which is the reason I got the extra cores/threads on the CPU.

 

[ocfguy]

Each drive uses 5-10 watts and about 25 watts at spin up. So anywhere from 100-500 watts. The CPU at idle is maybe 50 watts with 250 peak? I got a 1300 watt PSU to be sure I wouldn't be cutting it close. I'll have to see if my HBA allows staggered spin up, but I was planning for worst case.

I'm going to be running some VMs also which is the reason I got the extra cores/threads on the CPU.

Idle power consumption on Intel can be pretty low, like single digit

 

[oncdoc]

Each drive uses 5-10 watts and about 25 watts at spin up. So anywhere from 100-500 watts. The CPU at idle is maybe 50 watts with 250 peak? I got a 1300 watt PSU to be sure I wouldn't be cutting it close. I'll have to see if my HBA allows staggered spin up, but I was planning for worst case.

I'm going to be running some VMs also which is the reason I got the extra cores/threads on the CPU.

thanks for that. Im also thinking of putting together a fire breathing dragon system like yours. Maybe not as much. With the efficiency of newer chips the TDP is misleading probably.

 

[pd4ever]

thanks for that. Im also thinking of putting together a fire breathing dragon system like yours. Maybe not as much. With the efficiency of newer chips the TDP is misleading probably.

I'll grab some stats off the Kill-A-Watt meter next time I shutdown and can put it in-line. 10Gbe NIC just showed up today so shouldn't be too long.

 

[mr-rx-bs]

Asus W680 IPMI board showed up today. Can confirm compatibility with 4 x 32GB Hynix HMCG88MEBEA DDR5 DIMMs that were ordered from WiredZone. Installed TrueNAS and it is showing ECC is enabled. Had some issues with booting into TrueNAS originally with the IPMI card plugged in, so I unplugged it and will figure that out later.

Does ECC report being enabled in the BIOS? @Weapon reports that using the same DIMMs as you doesn't cause his BIOS to report it. Curious because I've got one on the way, along with the same DIMMs.

 

[Weapon]

I could never find a mention of ECC anywhere in the BIOS

 

[pd4ever]

Added the NIC and the Kill-A-Watt meter. Highest I saw it read during boot up was around 155 watts. The drive spin up didn't occur until TrueNAS was loading when it was about 120-125 watts. At idle the server is consuming between 105-108 watts. This is with a single hard drive, one NVME, GPU, HBA, and 10gbe NIC. Total wattage should be no problem. But I've read that the 5V rail can cause issues. My hot swap bays have a single SATA connector with an individual cable connected to a Peripheral port on the PSU for each hot swap bay. Does anyone know if those are using the 5V rail or the 12V rail?

When I get more drives installed I can do a test to see if they all spin up at once or if the HBA staggers by default.

Sorry I missed your question about the ECC in BIOS. I'll take a look next time I reboot. TrueNAS recognizes it as ECC on the Dashboard and is reporting ECC Correction Type as Single-bit ECC from dmidecode.

 

[pd4ever]

I added 4 more drives and went from 105-108 to 134-136 at idle with 5 hard drives total. So about 6-7 watts per drive idling. I think I saw the Kill-A-Watt peak around 220 watts during bootup. So it is likely that with 20 drives its going to be close to 600 watts or more during boot. I've been adding drives one at a time via hotswap so I can add a description field with the Bay number since the device name can change after a reboot. I will also eventually add serial number labels onto the drive doors when I'm using permanent drives. The ones I'm using now are just spare drives that I'm using for testing.

I went into the BIOS and clicked through everything. I'm not seeing anything specific to ECC either.

Could dmidecode be reporting the chip-level ECC thats built into the DDR5 specification as Single-bit ECC? This is my first system with ECC and with DDR5 so I don't fully understand the difference between the reporting of Single-bit vs Multi-bit ECC.

 

[ocfguy]

I added 4 more drives and went from 105-108 to 134-136 at idle with 5 hard drives total. So about 6-7 watts per drive idling. I think I saw the Kill-A-Watt peak around 220 watts during bootup. So it is likely that with 20 drives its going to be close to 600 watts or more during boot. I've been adding drives one at a time via hotswap so I can add a description field with the Bay number since the device name can change after a reboot. I will also eventually add serial number labels onto the drive doors when I'm using permanent drives. The ones I'm using now are just spare drives that I'm using for testing.

I went into the BIOS and clicked through everything. I'm not seeing anything specific to ECC either.

Could dmidecode be reporting the chip-level ECC thats built into the DDR5 specification as Single-bit ECC? This is my first system with ECC and with DDR5 so I don't fully understand the difference between the reporting of Single-bit vs Multi-bit ECC.

dmesg | grep -i EDAC

also does dmidecode say 64 bits or 72/80 bits? If it’s the latter then ECC should be enabled

 

[pd4ever]

dmesg | grep -i EDAC

also does dmidecode say 64 bits or 72/80 bits? If it’s the latter then ECC should be enabled

EDAC MC: Ver: 3.0.0


Total Width: 80 bits
Data Width: 64 bits

It looks like ECC is working as expected then?


Heres the full output for the first DIMM.
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.4.0 present.
# SMBIOS implementations newer than version 3.3.0 are not
# fully supported by this version of dmidecode.

Handle 0x004D, DMI type 16, 23 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: Single-bit ECC
Maximum Capacity: 256 GB
Error Information Handle: Not Provided
Number Of Devices: 4

Handle 0x004E, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x004D
Error Information Handle: Not Provided
Total Width: 80 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: Controller0-DIMM0
Bank Locator: BANK 0
Type: <OUT OF SPEC>
Type Detail: Synchronous
Speed: 4800 MT/s
Manufacturer: SK Hynix
Serial Number: --------
Asset Tag: ----------
Part Number: HMCG88MEBEA084N
Rank: 2
Configured Memory Speed: 4800 MT/s
Minimum Voltage: 1.1 V
Maximum Voltage: 1.1 V
Configured Voltage: 1.1 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 1, Hex 0xAD
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

 

[ocfguy]

Yeah I’d say so

 

[pd4ever]

Yeah I’d say so

OK great. Thanks!

 

[Lance Longreen]

Does anyone know if the Kingston version is any good ?

KSM48E40BD8KM-32HM

iit's Hynix based i see and seems to be only udimm ddr5 ecc in Europe in stock

 

[RolloZ170]

Does anyone know if the Kingston version is any good ?

have not tested but Kingston declares X13SAE, X13SAE-F and ASUS - Pro WS W680-ACE / IPMI as compatible.

 

[oncdoc]

I added 4 more drives and went from 105-108 to 134-136 at idle with 5 hard drives total. So about 6-7 watts per drive idling. I think I saw the Kill-A-Watt peak around 220 watts during bootup. So it is likely that with 20 drives its going to be close to 600 watts or more during boot. I've been adding drives one at a time via hotswap so I can add a description field with the Bay number since the device name can change after a reboot. I will also eventually add serial number labels onto the drive doors when I'm using permanent drives. The ones I'm using now are just spare drives that I'm using for testing.

I went into the BIOS and clicked through everything. I'm not seeing anything specific to ECC either.

Could dmidecode be reporting the chip-level ECC thats built into the DDR5 specification as Single-bit ECC? This is my first system with ECC and with DDR5 so I don't fully understand the difference between the reporting of Single-bit vs Multi-bit ECC.

Thank you for the testing. Very helpful

 

[BigBullion]

I have tested MTC20C2085S1EC48BA1R on a system with a Ryzen 7950X processor and a Gigabyte B650 Aero G motherboard.

Soon after pressing the DEL key and entering BIOS, I have quickly encountered bugs in the BIOS user interface. Generally in BIOS, there are menus containing lists of options that you can navigate with a keyboard and/or mouse. However, in this BIOS, when you click on an item, like item 3 on the list, item 2 is actually selected. If you click on item 4, item 3 is actually selected. When you click on an item, the item before the current option is actually being selected. So as a workaround, I have to click on the next item on the list if I want to select the current item. I guess this bug might be related to the fact that arrays start with "0" instead of "1".

I then ran MemTest86. It showed "N/A" for ECC in MemTest86. This showed up with the original BIOS version that came with the motherboard and the same "N/A" showed up after I updated the BIOS to the latest version with AGESA 1.0.0.4. I guess that AGESA 1.0.0.5 might support ECC.

It passed four passes of tests in MemTest86. I have attached a screenshot of dmidecode.

 

[diizzy]

@BigBullion
I'd be very interested in your results, Gigabyte themselves says "Add ECC DIMM support" in their previous BIOS update and it's unclear what that actually refers to. This post shows it as being detected at least using Memtest86+ 6.00 on AM5 (Asus mobo)

https://www.reddit.com/r/homelab/comments/zc0nb8/_/iyu9gae You probably also want to check the BIOS for any settings regarding ECC, from what I can tell it seems to be a bit random where menu options (if any) are located.

 

[BigBullion]

@BigBullion
I'd be very interested in your results, Gigabyte themselves says "Add ECC DIMM support" in their previous BIOS update and it's unclear what that actually refers to. This post shows it as being detected at least using Memtest86+ 6.00 on AM5 (Asus mobo)

https://www.reddit.com/r/homelab/comments/zc0nb8/_/iyu9gae You probably also want to check the BIOS for any settings regarding ECC, from what I can tell it seems to be a bit random where menu options (if any) are located.

I have fixed some errors in my earlier post. I am using the "MTC20C2085S1EC48BA1R" memory which has a similar name to the "MTC20C2085S1EC56BAZ" memory in their QVL.

I cannot find any BIOS option for ECC. Gigabyte's AM5 BIOS manual also does not mention it.

I don't think these screenshots from the post you have linked proves that ECC is supported... It lists it as "DDR5-4800 ECC" but that does not mean that it recognizes these modules as ECC nor does it mean that the motherboard detects or corrects errors. The "model name" of the memory might contain the words "DDR5-4800 ECC" and MemTest86+ could just be repeating its "model name".

ASUS does lists ECC support in their specs but this might be an error in their specs. Gigabyte and ASRock used to list ECC support in their specs and manuals but they eventually removed them in both their specs and manuals over time. It might be the case that ASUS just did not update them.

This post shows that their support technicians were saying that AM5 boards do not support ECC. The support technicians might be reading from a script that says that ECC is unsupported, but their website specs were not updated to match the scripts.

Also, their specifications page have other errors such as their claim (if I read it right) that HDMI 2.1 supports a max resolution/refresh rate of 4K60Hz (it's actually 8K60Hz). I have contacted a support technician to clarify its supported resolutions, after I showed him the error, and he said it is the reverse (4K60Hz for DP 1.4 and 8K60Hz for HDMI 2.1), but he was also wrong for saying that DP 1.4 supports a max resolution of 4K60Hz. I said that was also incorrect so he said that I should contact AMD for that information. Every other motherboard manufacturer lists 4K60Hz or 4K144Hz as its max resolution not 8K60Hz.

Not to mention that the pricing for their ASUS Pro WS W680-ACE motherboard might be an error. They released two boards of the same model, one with IPMI and one without IPMI. The board that includes the IPMI card is $80 cheaper than the board that does not include the IPMI card.

If you visit the IMPI page, and scroll down, you will also see a lot of broken images. Their website is very slow to load and their navigation bar only loads after the rest of the page is loaded. This is annoying since it pushes the content of the page down while you are reading it.

Even if ASUS mentions ECC support in their specs, it does not make ASUS legally obligated to provide support for ECC. From this page on their specs:

SPECIFICATIONS AND INFORMATION CONTAINED IN THIS MANUAL ARE FURNISHED FOR INFORMATIONAL USE ONLY, AND ARE SUBJECT TO CHANGE AT ANY TIME WITHOUT NOTICE, AND SHOULD NOT BE CONSTRUED AS A COMMITMENT BY ASUS. ASUS ASSUMES NO RESPONSIBILITY OR LIABILITY FOR ANY ERRORS OR INACCURACIES THAT MAY APPEAR IN THIS MANUAL, INCLUDING THE PRODUCTS AND SOFTWARE DESCRIBED IN IT.

 

[diizzy]

@BigBullion
Gigabyte's BIOS manual was created prior to the BIOS releases where they mention ECC so it makes sense that it does not mention ECC at all. ASRock otoh does however mention ECC in their BIOS manual however they don't seem to list what boards it applies to. It could also be that ECC gets silently enabled on your Gigabyte board which dmidecode or edac should tell. I'm also asking because you seem to have used memtest86 and not memtest86+ (different products).

Edit: I missed your screenshot :/

 

[ocfguy]

I have fixed some errors in my earlier post. I am using the "MTC20C2085S1EC48BA1R" memory which has a similar name to the "MTC20C2085S1EC56BAZ" memory in their QVL.

I cannot find any BIOS option for ECC. Gigabyte's AM5 BIOS manual also does not mention it.

I don't think these screenshots from the post you have linked proves that ECC is supported... It lists it as "DDR5-4800 ECC" but that does not mean that it recognizes these modules as ECC nor does it mean that the motherboard detects or corrects errors. The "model name" of the memory might contain the words "DDR5-4800 ECC" and MemTest86+ could just be repeating its "model name".

ASUS does lists ECC support in their specs but this might be an error in their specs. Gigabyte and ASRock used to list ECC support in their specs and manuals but they eventually removed them in both their specs and manuals over time. It might be the case that ASUS just did not update them.

This post shows that their support technicians are confused about ECC support for AM5 motherboards. The support technicians might be reading from a script that says that ECC is unsupported, but their website specs were not updated to match the scripts.

Also, their specifications page have other errors such as their claim (if I read it right) that HDMI 2.1 supports a max resolution/refresh rate of 4K60Hz (it's actually 8K60Hz). I have contacted a support technician to clarify its supported resolutions, after I showed him the error, and he said it is the reverse (4K60Hz for DP 1.4 and 8K60Hz for HDMI 2.1), but he was also wrong for saying that DP 1.4 supports a max resolution of 4K60Hz. I said that was also incorrect so he said that I should contact AMD for that information. Every other motherboard manufacturer lists 4K60Hz or 4K144Hz as its max resolution not 8K60Hz.

Not to mention that the pricing for their ASUS Pro WS W680-ACE motherboard might be an error. They released two boards of the same model, one with IPMI and one without IPMI. The board that includes the IPMI card is $80 cheaper than the board that does not include the IPMI card.

If you visit the IMPI page, and scroll down, you will also see a lot of broken images. Their website is very slow to load and their navigation bar only loads after the rest of the page is loaded. This is annoying since it pushes the content of the page down while you are reading it.

Even if ASUS mentions ECC support in their specs, it does not make ASUS legally obligated to provide support for ECC. From this page on their specs:

You can actually buy the W680 with IPMI: PROVANTAGE: Asus PROWSW680ACEIPMI Asus MB Pro WS W680-Ace IPMI W680 LGA1700 Max.128GB DDR5 ATX