LGA1366 - mismatched RAM across 2 CPUs? (Tyan S7012)

[TheBloke]

Hi all

I have a Tyan S7012 motherboard with 2 x 5670 CPUs (Westmere, 2.93ghz) and 18 RDIMM slots.

I started out with a total of 120GB RAM - 12 x 8GB + 6 x 4GB, arranged as 8, 8, 4GB in each of the 6 channels (3 channels per CPU * 2 CPUs).

I wanted to get in as much RAM as I could reasonably afford. I read that the mobo could go up to 288GB, meaning 18 x 16GB chips. What I didn't realise is that doing this required all chips to be dual rank!

So I bought 6 x 16GB quad rank chips. I thought I could have a config of: <16GB, 8GB, 8GB> in each of six total channels, giving me 192GB.

But it wouldn't boot, and that's when I checked again and realised the 288GB max was talking about all dual-rank chips, which in 16GB RDIMMS is stupidly expensive.

At first I thought I was going to be limited to 144GB, achieved from the following in each channel: <16GB, 8GB, empty>

But on re-reading the manual and trying various combinations, I've found that it works to have <16GB, 16GB, empty> in a channel. Therefore, taking the chips I have available, the maximum config I can achieve is as follows:

• CPU 1, three channels of: <16GB, 16GB, empty> - providing 96GB
• CPU 2, three channels of: <8GB, 8GB, 8GB> - providing 72GB
• Totalling 168GB

Not as good as the 192GB I thought I was going to get, but better than 144GB. (I must admit I don't understand why <16GB, 16GB, empty> works but <16GB, 8GB, 8GB> doesn't, given both are a total of 8 ranks. I guess that the number of slots also plays a part as well.)

But now I'm wondering if there are going to be performance problems with this config? What are the implications of having one CPU with 96GB and the other with 72GB? The OS can see and presumably use the total, 168GB, but will there be underlying performance issues?

This config is going to be the best I can do for now, because in order to balance it out and get the full 192GB I originally hoped for I would have to buy another 6 x 16GB. And although that would mean I could then sell all my other RDIMMS, it would still end up costing me a lot more than I planned to spend - 16GB chips are more than twice the cost of 8GBs.

In summary: is it theoretically OK to run different amounts of RAM across two CPUs? Should 168GB with mismatched RAM distribution be generally better than 144GB evenly distributed - or could the perf degredation of the uneven distribution make it so slow as to not be better? I should note that in both scenarios, the RAM will be running at 800mhz (my mobo does 1333mhz with 4 ranks used per channel, but drops straight to 800mhz if I use 6 or 8 ranks.)

Off the top of my head I can't think why it should cause a huge difference, but I don't understand the deep technical stuff going on on a CPU/mobo and so don't know if there can be subtle issues, or if perhaps a memory imbalance might put the chipset into some kind of reduced performance mode. There's already one obvious difference: for some reason it now takes twice as long for the server to reach the BIOS screen - maybe 50 seconds instead of 25 after hitting the power button before the monitor turns on.

(The use case is a Solaris 11.3 server providing a multi-TB ZFS pool and several VMs of varying types - Linux, Windows, and lots of Solaris zones. Home server but hopefully fairly heavily used.)

Thanks in advance

 

[TheBloke]

Answer: yes, it's fine. This is a NUMA motherboard and it's OK to have different amounts of RAM attached to each CPU. All CPUs can access all RAM.

Still not sure why the BIOS takes longer to boot. But reboots are rare, and especially so in Solaris which can reboot the kernel without rebooting the server as a whole. So it'll only come up occasionally.

 

[LukeP]

i have 18x8gb and was going to do the same. thanks for saving me the trouble. most 16gb are r4.

 

[TheBloke]

i have 18x8gb and was going to do the same. thanks for saving me the trouble. most 16gb are r4.

Cool, glad it helped someone! I started with 12 x 8GB + 6 x 4GB = 120GB, so I'm relatively happy with the 168GB I managed with 6 x 16GB chips. And it helped that I got them at an unusually good price, very close to twice the cost of an 8GB chip.

After realising the problem I did briefly look at getting six more, but the place I got them from was sold out and the standard price in on eBay UK is around £40 per RDIMM, compared to £15-£17 for an 8GB, so it's really not worth it knowing I could only go from 168GB to 192GB.

I did double check to confirm that this limitation applies to all LGA1366 motherboards, and of course it does - probably obvious as the memory controller is on-CPU, but I thought I'd best check.

I found a very detailed Fujitsu document that states that if a Quad Rank chip is installed, only two slots per channel can be used. And it describes 288GB as being achievable only with a mix of 16GB and 32GB chips! But maybe dual-rank 16GB chips weren't available when it was written.

Oh well, live and learn! I'm relatively happy with my 168GB total - at least I will be once I've sold off my excess chips such that it won't have cost quite as much money to achieve it

(In fact this same doc also answers the question I originally had - it describes supported asymmetrical memory configs, ie. having more RAM attached to one CPU than another. That was apparently a standard feature on some Fujitsu servers, which had more slots for one CPU than the other.)

 

[wildpig1234]

What are your experience with the S7012 MB and windows 10? does it support S3 sleep suspend to ram? run stable with win10?