about ram channel and bus speed limit

[jtaj]

I have plans to get 2nd generation cascade lake server, 6 channel with optane dimm, but with avx512 which uses a lot of bandwidth, ram speed will most likely be the bottleneck even with hexa channel (high core count CPU).

my question is, if I use optane dimm in junction with ram (optane storage mode, not using it as ram), would that lower ram performance since its all on same bus/connection? example, if I were to use a raid card with pcie 3.0 x8 connection that can exceed bus speed of 6.4GB/s I'd still get max of what 3.0 * 8 lanes can handle. but what about ram? all I see is the speed of ram but no where can I find what memory bus can actually provide.

would the bus be able to handle hexa channel ram running at full speed while also allow hexa channel optane storage at full speed, without cutting into each other's performance?

 

[i386]

Optane will be slower than dram/ram, but still faster than local storage (be it nvme, sas or sata devices).

From my understanding the memory channel will be using the "speed" of the slowest device in the memory channel for moving data to the cpu.

 

[jtaj]

Optane will be slower than dram/ram, but still faster than local storage (be it nvme, sas or sata devices).

From my understanding the memory channel will be using the "speed" of the slowest device in the memory channel for moving data to the cpu.

yes I get that, but my concern was if optane speed will interfere with ram speed.

say for example, I have ddr4 2666 ram in hexa channel getting me about 30GB/s read/write speed. PMEMs 2666mhz too in hexa channel gets about 25GB/s read/write. if I run the two in parallel, will the memory bus allow 55GB/s+ speed? or 30GB/s max due to ram? or worse, 25GB/s (i understand this is over simplified but thats what I am looking for, the speed limit of "memory bus", similar to my OP about PCIe 3.0 * 8 bus has a limit of 6.4GB/s).

I don't want to use PMEMs if it will hold back the ram, I can simply use PCIe storage for high sequential read/writes if thats the case.

 

[RolloZ170]

DCPMM Memory Configuration for Supermicro X11OPx/X11QPx/ X11DPx/X11SPx Motherboards

Operating Speed 1st Gen. DCPMMs operate at 2666 MHz DDR4 memory bus speed, and any installed 2933 MHz DDR4 memory will also operate at 2666 MHz.
may be different on other vendors.

 

[jtaj]

DCPMM Memory Configuration for Supermicro X11OPx/X11QPx/ X11DPx/X11SPx Motherboards

Operating Speed 1st Gen. DCPMMs operate at 2666 MHz DDR4 memory bus speed, and any installed 2933 MHz DDR4 memory will also operate at 2666 MHz.
may be different on other vendors.

I understand when optane paired up with ram, it'll run at the lowest frequency of whichever module is the lowest but thats wasn't my concern tho. I wish to find out if memory bus is capable to handle both at the sametime.

its not the same as if their performance in bandwidth will cut into each other. if you have optane + ram working together, could you help me test to see if your performance is lost when running both at the sametime?

one way to test this, maybe create a ram disk and run CDM with 20 samples of writes at say 2GB size, do the same for your optane disks and see if performance drop while running both simultaneously.

 

[RolloZ170]

if you have optane + ram working together, could you help me test to see if your performance is lost when running both at the sametime?

two RAM modules in the same channel(i.e. DIMMA1 and DIMM_A2) do not work in parallel. one can decode while the other transfers but must wait, there is only one data-addressbus in the channel.

 

[jtaj]

two RAM modules in the same channel(i.e. DIMMA1 and DIMM_A2) do not work in parallel. one can decode while the other transfers but must wait, there is only one data-addressbus in the channel.

I see, so basically when optane dimm is in use, ram must wait for optane dimm to finish transfer?

if thats the case, what is the best setup for optane dimms then? 6 channel of rams, and also for dimm slot 2, 6 channels of optane seems a kill joy if they can't be used at the same time.

 

[RolloZ170]

I see, so basically when optane dimm is in use, ram must wait for optane dimm to finish transfer?

if RAM must wait for RAM, RAM must wait for PMEM too.
RAM in slot 1 can't be used in same time with RAM slot 2, PMEM makes no difference there.

 

[jtaj]

if RAM must wait for RAM, RAM must wait for PMEM too.
RAM in slot 1 can't be used in same time with RAM slot 2, PMEM makes no difference there.

ugh thats a big let down. I supppose its better to go ram + nvme SSD combo rather than ram + optane dimm and using pmem as storage. though without actual hardware to test the workload i'd have 0 ideas, this is of course assuming both are underload at the sametime which may not be true at all.

edit: so to prevent them from interfering one another I'd have to choose say 4 or 5 channel of ram and have optane dimm populate on remaining 2 or 1 channel for them not to interfere each other is that correct?