RAM capacity with limited DIMM slots on mini-ITX

[logan893]

I'm looking at buying a mini-ITX server based on the ASUS P10S-I which sports 2 DIMM slots and an LGA1151 socket for CPUs like Xeon E3-1275v6.

The motherboard capacity says max 32GB RAM (16GB per module), since there are only 2 DIMMS instead of 4. The official CPU support according to Intel is 64GB (2 channels and 2 DIMMs per channel).

Not sure how reliable it is but it seems possible that Intel does not necessarily claim the absolute maximum capacity in these specs, but it may be more akin to what was available/common and that they have validated. One example on the low-end is the Celeron J4125 that Intel says supports max 8GB (2 channels), but has been shown to work with 16GB or perhaps even 32GB total RAM. (Source)

In this case, the CPU claims the total amount is supported, it is merely the capacity per DIMM that would be higher than what the motherboard claims is supported, since the capacity per DIMM required wasn't readily available when these were released.

32GB DDR4 ECC unbuffered RAM modules probably didn't exist yet in 2017, or at least were very rare.

What are the chances that a setup with a Xeon E3-1200v6-series CPU on a 2-DIMM-slot mini-ITX mobo like this would work well with 2x32GB RAM for a total of 64GB?

 

[RolloZ170]

Xeon E3-1275v6

the E3-1275 v6 supports up to 8GB 16GB per Rank.
with 2x 32GB 4Rx8 UDIMM you would get 64GB total. unfortunaly 32GB quad raked UDIMM modules does'nt exist (AFAIK)

 

[logan893]

I believe what you mention is also what they list e.g. in the table "Supported DDR4 ECC UDIMM Module Configurations" on page 21 in the datasheet, but perhaps it may be because they have the "product segmentation limit" for the CPU at 64GB, and there were no 16GB single rank modules available (or 32GB dual rank) at that time.

I suppose I'm wondering if this is a hard limit, or could it be what has been "verified to work" but more might be possible...
May I ask where your information of max 8GB per Rank comes from?

 

[Wasmachineman_NL]

Kaby Lake will support 32GB (SO)DIMMs no problem.

 

[logan893]

Kaby Lake will support 32GB (SO)DIMMs no problem.

That is promising! I may just go ahead and give it a try.

 

[logan893]

For some additional confirmation bias, I found this German reseller with a 100% compatibility guarantee for what seems to be their own 32GB modules (and 2x32GB for 64GB total) with the ASUS P10S-I motherboard. The 32GB modules listed are non-ECC, though. Their ECC UDIMMs are at most 16GB each.

32GB Arbeitsspeicher DDR4 für Asus P10S-I RAM UDIMM

Arbeitsspeicher Asus P10S-I Der RAM Speicher erweitert die Speicherkapazität des Asus P10S-I um 32GB. Der Arbeitsspeicher des Asus P10S-I kann…

www.speicher.de

 

[logan893]

Initial results are in.

1x32GB works. 1x32GB + 1x4GB works.

There's possibly something odd going on, though. Not sure if it's significant enough to be a deal-breaker though.

When I run memtest86, I get 8 "ECC Error Detected" (corrected errors) in the first two or three tests executed on the first pass. Typically one from test 0, six or seven from test 1, and sometimes the final error from test 2. Not always the exact same rows, but always the same rank 0 and bank 0, and row in the 10000-1EE00 range. Mostly columns 0 and 8, but sometimes others like C, 10, 18. So far there are additional errors reported beyond these initial 8, until after a system reboot. Re-running the same tests without restarting passes without further errors detected.

Doesn't matter which channel I slot the 32GB stick into, the 8 errors reported move with the stick.

I've ordered a 2nd identical 32GB stick to see where that gets me. That should hopefully also tell me whether the issue I'm having is related to the RAM stick, or some general jankiness with this RAM+mobo combination. Will need to run some dummy load for a while and monitor for MCEs.

 

[RolloZ170]

When I run memtest86, I get 8 "ECC Error Detected" (corrected errors) in the first two or three tests executed on the first pass

you have non ECC. turn OFF ECC error polling.
32GB + 4GB requires support from the memory controller(Flex-Memory), not supported by all SKUs.

 

[logan893]

you have non ECC. turn OFF ECC error polling.
32GB + 4GB requires support from the memory controller(Flex-Memory), not supported by all SKUs.

I got this exact same behavior with only the single stick of 32GB DDR4 ECC memory. Happens almost every time. Only once after a fresh reboot have I not seen this same string of 8-9 errors.

I have though been running some more tests while mucking about in the BIOS to see if I can get this to go away but so far there's even more strange stuff going on...

 

[logan893]

you have non ECC. turn OFF ECC error polling.
32GB + 4GB requires support from the memory controller(Flex-Memory), not supported by all SKUs.

Strange behavior I'm seeing do not seem to be due to any BIOS changes, so more likely it's down to platform limitations or some combination with this DIMM.

They are related only to using the 32GB stick, on its own or with a 4GB stick doesn't seem to matter much.

The PC currently has Proxmox installed, and a single VM with pfSense. Behavior is the same both with a single 32GB stick of RAM, or with a 32+4 combo. The system often manages to fully boot (twice so far have I seen a kernel panic during boot) and begins the automatic start-up of the pfSense VM. The system crashes with a reset relatively quickly, usually within the first 2-3 minutes.

I installed edac-utils and it prints detection of uncorrectable errors in the syslog. It's odd that memtest86 doesn't show any actual errors (beyond the 8-9 correctable ones initially) even after multiple passes.

I'm quite limited with what I can configure in BIOS of this ASUS motherboard, and I don't have any other system to validate the DIMM in. I have another 32GB stick on the way by post but that may not behave any differently. I guess I'll keep them for a potential Ryzen system in the near future.

I have had minor successful result so far by reducing "Maximum Memory Frequency" all the way down to 1067 (Auto selects the maximum supported by the CPU, 2400, while the DIMM itself supports up to 3200 MHz.) At 1867 this seems to remove the correctable errors in memtest86. However, it doesn't improve stability in Proxmox, which still crashes.

Disabling error reporting (WHEA) doesn't seem wise if there are actually memory errors, but it does seem to be the only thing which keeps Proxmox from crashing at least in the short term. Firing up memtest86 in a VM throws me a similar amount of 8-9 errors in the host syslog as when running memtest86 on baremetal, but they show as Uncorrectable Errors. Yet, there are no errors in the memtest86 VM output.

Here's the output from running memtest86 in a VM, with WHEA switched off in host BIOS. The output doesn't seem to contain much of any useful information, and perhaps it is not accurate considering memtest86 shows no errors in the VM.

Mar 23 22:50:13 pmox44 kernel: [  197.974511] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:8)
Mar 23 22:50:19 pmox44 kernel: [  204.114847] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#1channel#0 (csrow:1 channel:0 page:0x0 offset:0x0 grain:8)
Mar 23 22:50:23 pmox44 kernel: [  208.210943] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:8)
Mar 23 22:50:26 pmox44 kernel: [  211.282932] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:8)
Mar 23 22:50:27 pmox44 kernel: [  212.306789] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:8)
Mar 23 22:50:30 pmox44 kernel: [  215.378817] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:8)
Mar 23 22:50:32 pmox44 kernel: [  217.427114] EDAC MC0: 1 UE UE overwrote CE on any memory ( page:0x0 offset:0x0 grain:8)
Mar 23 22:50:35 pmox44 kernel: [  220.498881] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:8)
Mar 23 22:50:37 pmox44 kernel: [  221.523136] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#1channel#0 (csrow:1 channel:0 page:0x0 offset:0x0 grain:8)
Mar 23 22:50:38 pmox44 kernel: [  222.547472] EDAC MC0: 1 UE ie31200 UE on mc#0csrow#0channel#0 (csrow:0 channel:0 page:0x0 offset:0x0 grain:8)

Is there any use and way to inject ECC errors with a free utility, and would it be worthwhile to test? I don't have access to memtest86 pro.

 

[RolloZ170]

if i had that problems with a mobo cpu RAM combo, i would throw all out of the (open)window. time is money.

 

[logan893]

If this was an officially supported combination, and anything more than a hobby project, then I'd be more inclined to agree.

 

[logan893]

2nd stick is now installed, and this seemingly "false positive" initial error reporting remains quite consistent.

Now memtest86 running on baremetal reports around 26 correctable errors (in pairs, 13 per stick) after a cold boot. After triggering these errors and issuing a soft restart (ctrl+alt+del) the subsequent memtest86 runs report no errors. Same with booting into proxmox, first cold boot shows some UE errors deteceted by EDAC, and subsequent soft reboots show no errors. memtest86 in a VM reports no actual memory errors even when the host system reports these initial errors (and it's only when memory is accessed that the errors are reported, so it must be from the memtest86 VM.)

Something additionally odd is that for a while during these tests, the baremetal boot of memtest86 was VERY slow. Took over 3 minutes, with a whole minute before anything was shown on screen and long pauses between each initialization step. Once in the actual GUI, everything ran fine. For a while when running with 32+4 GB, and now with 2x32GB, the boot into memtest86 is very quick. No clue what that slowdown may be caused by. This was reproduced even with the one or two 4GB sticks of RAM, yet for a while went away when I had 32+4 GB installed, so perhaps there was some memory training hiccup or interference unrelated to memory (possibly USB) bogging things down. Memory errors are not impacted by the fast or slow start-up of memtest, and proxmox was always quick to boot.

Anyway, for now it seems good enough and I'll keep stress testing for a while; perhaps look for some way to test ECC Error Injection. The CPU supports it, but I have my doubt about the motherboard and chipset and I can't find any info about it.

 

[semuel]

Anyway, for now it seems good enough and I'll keep stress testing for a while; perhaps look for some way to test ECC Error Injection. The CPU supports it, but I have my doubt about the motherboard and chipset and I can't find any info about it.

Dear Logan893, can you please confirm that P10S-I works well with 64GB ram ?

I am in the same shoes right now: having the server with p10s-i itx running out of ram. So I try to keep the server without changing the motherboard.

 

[logan893]

Been running with the 64 GB (2x32) since my last post here, and it's seemingly rock solid beyond those initial (I assume false positive) errors on boot. I can't speak for any impact to actual ECC behavior (detection or correction), but so far I have not had any problems.

I run Proxmox with ZFS on two SSDs and among other things one VM with TrueNAS Core allocated 50% of the RAM; no reported problems with the data on the drives neither in Proxmox nor the TrueNAS VM.

Only trouble I had initially was related to the PCIe HBA, where I had to cover pins B5 and B6 or else only a single stick of RAM would work. Something to do with an SMBus conflict, not really related to the quirks I described in this thread.

I cut a small piece of Kapton tape to cover those pins and that has worked well.

My references:

SOLVED - X11ssm-f DIMMB2 disappears after adding PCIe SLOG

I hope someone has seen this before. I have a typical x11ssm-f setup with FreeNAS on bare metal. I recently migrated from my ESXi to a simpler setup. Everything has been rock solid for at least 3 years. I recently picked up a DC P3700 HHHL card to use as a SLOG. I loaded the card into...

www.truenas.com

 

[Andriy Gapon]

Just want to share my experience using single-rank 16GB (ECC) UDIMMs with X11SSM-F motherboard and a E3 v6 processor.
And I must say that I surprised that people here got 32GB sticks working.

For starters, X11SSM-F manual says that 16GB DIMMs must be dual-rank.
I thought that maybe that information was outdated because single-rank 16GB DIMMs did not exist at the time.
But it looks like the motherboard does have that limitation.

I also read the specification for Xeon E3 1200 v6 processors and, if I am reading it correctly, it also says that 16GB DDR4 DIMMs must be dual rank, see tables in section 2.1.1.2.

I actually tried a single rank stick and the motherboard did recognize it and its full capacity.
But access to the upper 8GB of the stick was extremely slow and the IPMI log were getting filled with ECC checksum errors (probably detecting and logging those errors is what caused the slowness).

I suspect that the hardware lacks an address line to address full 16GB in a single rank, so accesses to the upper half probably go to a wrong place.

But I am open to the idea that it's the motherboard that has the limitation, not the processor.
Maybe there is also a difference between ECC and non-ECC UDIMM support.

 

[RolloZ170]

I suspect that the hardware lacks an address line to address full 16GB in a single rank, so accesses to the upper half probably go to a wrong place.

DRAM chip densitiy issue. 8/16Gbit parts.
1) BIOS don't know about
2) cpu lacks the address line.
3) address line not wired from socket to DIMM slots

Maybe there is also a difference between ECC and non-ECC UDIMM support.

just one chip more for ECC bits. (64 -> 72 bits)