LGA 1700 Alder Lake "Servers"

[adman_c]

That is the exact same part I am using in my workstation with Linux, and it reports as 72 bits. You are using Windows and using powershell or what tool to look at the width?

This is on Proxmox using the 6.1.14 kernel

dmidecode -t memory
Handle 0x004F, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x004D
        Error Information Handle: Not Provided
        Total Width: 80 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: None
        Locator: Controller0-DIMM1
        Bank Locator: BANK 0
        Type: <OUT OF SPEC>
        Type Detail: Synchronous
        Speed: 4800 MT/s
        Manufacturer: Micron Technology
        Serial Number: 
        Asset Tag: 9876543210
        Part Number: MTC20C2085S1EC48BA1
        Rank: 2
        Configured Memory Speed: 4800 MT/s
        Minimum Voltage: 1.1 V
        Maximum Voltage: 1.1 V
        Configured Voltage: 1.1 V
        Memory Technology: DRAM
        Memory Operating Mode Capability: Volatile memory
        Firmware Version: Not Specified
        Module Manufacturer ID: Bank 1, Hex 0x2C
        Module Product ID: Unknown
        Memory Subsystem Controller Manufacturer ID: Unknown
        Memory Subsystem Controller Product ID: Unknown
        Non-Volatile Size: None
        Volatile Size: 32 GB
        Cache Size: None
        Logical Size: None

 

[RolloZ170]

To me that is enough documentation to show that both x72 and x80 ECC DDR5 variants are in the wild.

another inconsistency:
RDIMMs with 72 bit begin with MTC09 or MTC18
RDIMMs with 80 bit begin with MTC10 or MTC20
but ECC UDIMMs with 72 bit begin with MTC10 or MTC20 (should be 09 or 18 thought)

 

[heromode]

I'm gonna say something dumb here but could the 2x4 bit difference have anything to do with amount of ram channels, and dimm's per channel?

 

[RolloZ170]

I'm gonna say something dumb here but could the 2x4 bit difference have anything to do with amount of ram channels, and dimm's per channel?

no. 72 bit just lacks half of the ECC bits. maybe 72 bit is single bit ECC corr. and 80 bit is multi bit.

 

[adman_c]

Weird that they’d put multi bit in a UDIMM when most systems that use UDIMMs are only capable of single bit correction (right?). My W680 board says single bit ECC in dmidecode.

 

[RolloZ170]

Weird that they’d put multi bit in a UDIMM when most systems that use UDIMMs are only capable of single bit correction (right?)

unclear: lga1700 socket has 2x 40 bits data lines. what can we do with double the ECC bits ?

 

[James C. Owens]

Yes... there have to be 2x40 data lines to accommodate both x72 and x80 ECC configs apparently. I think both forms of ECC are working if you have the correct CPU. What is not clear is what the difference in implementation is in sideband ECC between the 2x36 and the 2x40. Also interesting is that Windows (I think based on the post above from @adman_c) is reporting x80 for the same part number that I have, where Linux is reporting x72, and the Micron specs say x72. It is hard to say at this point, but it is possible that the Windows API is simply not right, and assumes that all DDR5 ECC is 2x40 = 80.

It would be nice for Intel/AMD and the memory manufacturers to straighten this out a little better so it was more clear for those of us that worry about ECC.

 

[James C. Owens]

And I agree with all of the earlier posts where it was stated that ECC as implemented in single module (i.e. not chipkill stuff) is Single-bit Error Correction Multi-bit Error Detection. Perhaps the extra 4 bits on each channel allows for some Multi-Bit Correction too under certain scenarios? It is unclear. I would think the x72 ECC config would provide exactly the same ECC capabilities as with DDR4, as I suspect the same Hamming codes and syndrome tables would be used.

I still have not been able to find a detailed document that provides real technical information on how sideband ECC (not on-die) in DDR5 is implemented.

 

[James C. Owens]

As a further note that may be exactly why there are x72 width ECC DDR5 modules in the wild. It may be easier (i.e. cheaper) for the manufacturers to use the same Hamming codes and syndrome tables that were used for DDR4 ECC RAM before rather than do new ones. The 64 dollar question is whether there are any true 80 bit width unbuffered ECC DDR5 modules out there. The only 80 bit ones I saw were on the Micron site, and they were registered.

 

[James C. Owens]

Weird that they’d put multi bit in a UDIMM when most systems that use UDIMMs are only capable of single bit correction (right?). My W680 board says single bit ECC in dmidecode.

Ah you are using Linux not Windows then. What kernel are you running?

 

[James C. Owens]

This is on Proxmox using the 6.1.14 kernel

dmidecode -t memory
Handle 0x004F, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x004D
        Error Information Handle: Not Provided
        Total Width: 80 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: None
        Locator: Controller0-DIMM1
        Bank Locator: BANK 0
        Type: <OUT OF SPEC>
        Type Detail: Synchronous
        Speed: 4800 MT/s
        Manufacturer: Micron Technology
        Serial Number:
        Asset Tag: 9876543210
        Part Number: MTC20C2085S1EC48BA1
        Rank: 2
        Configured Memory Speed: 4800 MT/s
        Minimum Voltage: 1.1 V
        Maximum Voltage: 1.1 V
        Configured Voltage: 1.1 V
        Memory Technology: DRAM
        Memory Operating Mode Capability: Volatile memory
        Firmware Version: Not Specified
        Module Manufacturer ID: Bank 1, Hex 0x2C
        Module Product ID: Unknown
        Memory Subsystem Controller Manufacturer ID: Unknown
        Memory Subsystem Controller Product ID: Unknown
        Non-Volatile Size: None
        Volatile Size: 32 GB
        Cache Size: None
        Logical Size: None

Here is mine, running on OpenSUSE Leap 15.4 (with kernel 5.14.21-150400.24.46-default).


Handle 0x0020, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x001C
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMMB2
Bank Locator: P0_Node0_Channel1_Dimm1
Type: DDR5
Type Detail: Synchronous Unbuffered (Unregistered)
Speed: 4800 MT/s
Manufacturer: Micron Technology
Serial Number: S802C0F22453C3DB3E1
Asset Tag: 9876543210
Part Number: MTC20C2085S1EC48BA1
Rank: 2
Configured Memory Speed: 3600 MT/s
Minimum Voltage: 1.1 V
Maximum Voltage: 1.1 V
Configured Voltage: 1.1 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 1, Hex 0x2C
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

Another interesting difference between yours and mine is that mine are at 3600 MT/s, whereas yours are at 4800. I left all of the Supermicro memory settings at auto, and I have all four slots filled, which may be reducing the default memory speed. (IIRC the standard speeds slow down if all banks are filled.)

Do you have rasdaemon running, and if so, what does the output of ras-mc-ctl say?

 

[James C. Owens]

I just answered my own question on the speed issue...

See Intel Alder Lake RAM Guide: Picking Between DDR4 and DDR5 | Tom's Hardware (tomshardware.com) .

And specifically:

DDR5 Frequency (MT/s)	Slots Per Channel (SPC)	DIMMs Populated Per Channel (DPC)	Rank (s)
4,000	2 SPC	2 DPC	1 R
3,600	2 SPC	2 DPC	2 R
4,400	2 SPC	1 DPC	1 R
4,400	2 SPC	1 DPC	2 R
4,800	1 SPC	1 DPC	1 R
4,800	1 SPC	1 DPC	2 R

 

[James C. Owens]

I need the 128 GB of ram for what I am doing, so ... 3600 it is. I do not want to play with fire by trying to run it above spec for fully populated.

 

[RolloZ170]

can someone check the P/N of the chips on the ECC UDIMM module ?
80 bit is x8 | x8 x8 x8 x8 || x8 x8 x8 x8 | x8
72 bit should be x4 | x8 x8 x8 x8 || x8 x8 x8 x8 | x4

 

[James C. Owens]

That is a good idea. I took close-up pictures of the modules, but only the side that has the serial number label, and it obscures the parts. I will have to find a time to shut the machine down and take out a module to do this. Right now it is busy...

 

[saf1]

If it helps any below is my output on my Ubuntu 20.04 distro:

dmidecode -t memory
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.5.0 present.

Handle 0x0020, DMI type 16, 23 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: Single-bit ECC
Maximum Capacity: 128 GB
Error Information Handle: Not Provided
Number Of Devices: 4

Handle 0x0022, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0020
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMMA2
Bank Locator: P0_Node0_Channel0_Dimm1
Type: DDR5
Type Detail: Synchronous Unbuffered (Unregistered)
Speed: 4800 MT/s
Manufacturer: SK Hynix
Serial Number:
Asset Tag:
Part Number: HMCG88MEBEA084N
Rank: 2
Configured Memory Speed: 4400 MT/s
Minimum Voltage: 1.1 V
Maximum Voltage: 1.1 V
Configured Voltage: 1.1 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 1, Hex 0xAD
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

Handle 0x0024, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0020
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMMB2
Bank Locator: P0_Node0_Channel1_Dimm1
Type: DDR5
Type Detail: Synchronous Unbuffered (Unregistered)
Speed: 4800 MT/s
Manufacturer: SK Hynix
Serial Number:
Asset Tag:
Part Number: HMCG88MEBEA081N
Rank: 2
Configured Memory Speed: 4400 MT/s
Minimum Voltage: 1.1 V
Maximum Voltage: 1.1 V
Configured Voltage: 1.1 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 1, Hex 0xAD
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

 

[Alex_T0000]

Could you confirm the CPU you bought in the end - I thought from reading an earlier post you had gone for an i5-12500? Though I guess might have gotten confused after reading 25+ pages of comments

Just asking as was wondering which CPU the power consumption numbers you posted are based on.




Also is there a model number or any other identifying info on the IPMI card? Where I am I can't seem to get the IPMI version, but have found a card available separately described as -

"
Asus Ipmi Expansion Card W/ Dedicated Ethernet Controller Vga Port Pcie 3.0 X1 &
ASUS IPMI expansion card with dedicated Ethernet controller, VGA port, PCIe 3.0 x1 interface and ASPEED AST2600A3 chipset to provide the intelligence for IPMI architectures "

Hey there, sorry for the late reply!! it's been some crazy stuff going on in life, so haven't had much time to check the forums!

So I ended up going with a Core i5-13600K, and the load numbers that I was basing on, were based on what my UPS unit is indicating, however, there is also an Asustor NAS on there with 4 x 10TB disks, I would expect the total power draw on that to be 22-40w of what the UPS is indicating.

As a note, I have the following devices connected to that UPS.

The system, that contains the WS680-ACE IPMI, an 9211-8i HBA in IT mode, the Core i5 13600K - CPU, 64GB DDR5-ECC UDIMM, 1TB PCIE v4 SSD, 256GB PCI v4 SSD, the following drives:

WDC WD140EDGZ-11B1PA0
WDC WD140EDGZ-11B1PA0
WDC WD120EDAZ-11F3RA0
WDC WD120EDAZ-11F3RA0
ST18000NE000 (18TB IronWolf PRO)
ST18000NE000 (18TB IronWolf PRO)
WDC WD40EFRX-68WT0N0
WDC WD40EFRX-68WT0N0
WDC WD50EFRX-68MYMN1
WDC WD50EFRX-68MYMN1
WDC WD80EFZX-68UW8N0
WD80EFAX-68KNBN0
WD80EFAX-68KNBN0

Plus connected via USB 3 in their own enclosures powered up.
ST8000AS0002-1NA17Z (8TB Archive Series Seagate (SMR))
ST8000DM004-2CX188 (8TB Barracuda)

The NAS has:
WD101EFBX-68B0AN0 (WD 10TB Red Plus)
WD101EFBX-68B0AN0 (WD 10TB Red Plus)
WDC WD100EFAX-68LHPN0 (WD 10TB Red Plus)
WDC WD100EFAX-68LHPN0 (WD 10TB Red Plus)

So if we look at the drives spinning.. it's an average of 4.5w per drive (just for arguments sake) - 85.5 w total power budget on spinning rust.

This leaves the rest of the system (MB/CPU/SAS PCI controller, 2 NVM-E SSDs) and the Asustor NAS consuming 77.5 watts.

Note, I took this picture just now with the system running in maximum power savings, as I don't see the system seeing much action these days, so I changed the power profile in windows to be as such, but when all drives are running and the system is hosting the 2 servers (Valheim / Minecraft) it's 186 watts, meaning a power draw of 97.5 watts for (MB/CPU/SAS PCI controller, 2 NVM-E SSDs) and the Asustor NAS at the socket. I would think the NAS consumes probably 8-12 watts on it's own.

PL1 and PL2 settings for the CPU are set to Intel stock numbers - 125w and 181w respectively, as per Intel ARK.

So overall, not bad I think... I hope this information above helps!


RE: IPMI - I don't know, but I would ensure that the BIOS has support for it, as the WS 680ACE board does - it has a dedicated menu for the IPMI card and it can initialize the card and manage it from the BIOS. Posted the ASUS compatible model link for you - I don't think this card will work on any other board or even other non-specific ASUS board models - since the BIOS has a page dedicated to it. https://dlcdnets.asus.com/pub/ASUS/mb/Add-on_card/IPMI_EXPANSION_CARD/Compatible_Model_List.pdf?model=ipmi expansion card

 

[Alex_T0000]

Can confirm the PL1/PL2 can be set in bios. I had issues with mine continually going over what I set and had to disable the Asus feature that dynamically changes PL's. I'm running a i5-12500T at stock wattage and my server pulls about 90watts at low load. That's with 12HDD's, 3x m.2's, 2x sata SSD's and 2x HBA's in the case.

I have the IPMI version and struggled to find any identifying marks that told me much about the card. All I can really say is it's a AS2600 based board. It works well so far for me. Wiring is kind of clunky, but it's to be expected.

The only real issue I've had is getting the iGPU to show up. At first it was fine, but I stopped getting output from the board HDMI at one point and the iGPU fell off as a device. I found the server was only picking up the IPMI card as a GPU and ignoring the iGPU even with the iGPU set as primary display output in the BIOS. Ended up enabling the feature allowing the iGPU and dGPU to both be used simultaneously for displays deep in the advanced settings to resolve the issue.

To ensure the Core i5's GPU is running all the time and never shuts down, I did this instead of changing the BIOS - got a DisplayPort Emulator, which fools the GPU into thinking a monitor is plugged in.

EVanlak disport Display Emulator 3rd Generation Premium Aluminum DP Dummy Plug take Advantage of a high end Graphics Card remotely fit Headless 2560x1600@60Hz : Amazon.ca: Electronics

 

[TheDragon44]

Kingston claims their ECC sticks are compatible with ASUS W680-ACE IPMI.

I ordered some of their 32GB sticks to test.

KSM48E40BD8KM-32HM

View attachment 26786

Did the Kingston memory you ordered work ok in the asus pro w680-ace ipmi, and show up as ECC enabled?

I tried to get the Hyinx modules but couldn't find them to buy anywhere in my country, the Kingston modules seem to be widely available though

 

[ddr5ecc]

I have two KSM48E40BD8KM-32HM and the Asus WS W680 here.

and a Core i7 13700K

And it isn't running.

If I put only one DIMM into the Socket it is booting but with ridiculous fast RAM timings(normal for fast DDR5-RAM but too fast for DDR5-ECC).

One DIMM is even memtest stable.

But if you are putting 1 DIMM per channel into the system it isn't even booting.

I updated to the newest firmware and it isn't helping.

I think I am going to send it back.

I have a X13SAE+13700K+Kingston-ECC running with conservative RAM settings. Everything is running rock stable with 1DPC.

Currently with the Asus Board I try to set the RAM settings CL tRCD .... manually but so far no luck.


This is my X13SAE config