Help needed populating Optane DC modules (DCPMM, PMem 100 modules)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
M

metricv

New Member
Jun 8, 2024
5
4
3
I am having some difficulty configuring my Dell 7920 Tower with Optane DCPMM memory. The CPU I'm using is Xeon 8259CL (2nd Gen Cascade Lake, with L-suffix).

Here are the things I have:
- 4 x Samsung 64GB DDR4-2666 LRDIMM, M386A8K40BM2-CTD. Denoted as D
- 4 x Intel 512GB Optane PMem 100, NMA1XXD512GPS. Denoted as P.
All these modules are verified working as they show up in the BIOS and via ipmctl show -topology.

OS is Truenas Scale, Linux truenas 6.6.29-production+truenas #1 SMP PREEMPT_DYNAMIC Wed May 29 15:05:57 UTC 2024 x86_64 GNU/Linux

Here are the configurations I've tried. (Let's not talk about bandwidth for now. I know some configs are dumb, but let's try to get them working first.)

Original Configuration:
Code: 
+-----------------------------------------------+
|                      CPU0                     |
+-----------------------------------------------+
|          IMC1         |          IMC0         |
|  Ch5  |  Ch4. |  Ch3. |  Ch0. |  Ch1. |  Ch2  |
| 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
| D | P | D |   |   |   |   |   |   |   |   |   |

+-----------------------------------------------+
|                      CPU1                     |
+-----------------------------------------------+
|          IMC1         |          IMC0         |
|  Ch5  |  Ch4. |  Ch3. |  Ch0. |  Ch1. |  Ch2  |
| 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
|   |   |   |   |   |   |   |   |   | D | P | D |
This is the configuration before I added two more 512GB Optane PMem modules. BIOS shows 1TB memory installed, and 988G shows up in the OS, everything is fine.

Adding two PMem modules, attempt 1
Code: 
+-----------------------------------------------+
|                      CPU0                     |
+-----------------------------------------------+
|          IMC1         |          IMC0         |
|  Ch5  |  Ch4. |  Ch3. |  Ch0. |  Ch1. |  Ch2  |
| 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
| D | P | D | P |   |   |   |   |   |   |   |   |

+-----------------------------------------------+
|                      CPU1                     |
+-----------------------------------------------+
|          IMC1         |          IMC0         |
|  Ch5  |  Ch4. |  Ch3. |  Ch0. |  Ch1. |  Ch2  |
| 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
|   |   |   |   |   |   |   |   | P | D | P | D |
After I did this, all 8 memory modules show up in the BIOS correctly, and BIOS reported 2TB memory installed. However, only 1TB shows up in the system, and usable memory is still 988G.

Adding two PMem modules, attempt 2
Code: 
+-----------------------------------------------+
|                      CPU0                     |
+-----------------------------------------------+
|          IMC1         |          IMC0         |
|  Ch5  |  Ch4. |  Ch3. |  Ch0. |  Ch1. |  Ch2  |
| 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
| P |   | D |   |   |   |   |   |   | D |   | P |

+-----------------------------------------------+
|                      CPU1                     |
+-----------------------------------------------+
|          IMC1         |          IMC0         |
|  Ch5  |  Ch4. |  Ch3. |  Ch0. |  Ch1. |  Ch2  |
| 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
| P |   | D |   |   |   |   |   |   | D |   | P |
This is actually similar to the recommended way of populating PMem modules in T7920 owners' menu. However, under this configuration, BIOS shows 1291392MB (~1TB + 256GB) memory installed, and OS reports only 256GB memory usable. ipmctl shows that no PMem is mapped.

Can someone point out what is wrong with my configuration? My goal is to have all 2TB memory availalble and usable by the OS.

Thanks everyone for stopping by!
 
J

joerambo

New Member
Aug 30, 2023
17
4
3
metricv said:
After I did this, all 8 memory modules show up in the BIOS correctly, and BIOS reported 2TB memory installed. However, only 1TB shows up in the system, and usable memory is still 988G.
Click to expand...
Any settings in BIOS to control split between PMEM and "mixed" space? Is ipmctl showing them as connected in the OS ( as pmem ).
 
M

metricv

New Member
Jun 8, 2024
5
4
3
joerambo said:
Any settings in BIOS to control split between PMEM and "mixed" space? Is ipmctl showing them as connected in the OS ( as pmem ).
Click to expand...
T7920 only supports Optane in memory mode (no App-Direct mode). I don't recall such a configuration in BIOS; will check later.

Here's how it appeared in the OS
Code: 
root@truenas[/home/admin]# ipmctl show -topology
DimmID | MemoryType                  | Capacity    | PhysicalID| DeviceLocator
================================================================================
0x0011 | Logical Non-Volatile Device | 502.563 GiB | 0x0011    | DIMM9_CPU0
0x0021 | Logical Non-Volatile Device | 502.563 GiB | 0x0015    | DIMM7_CPU0
0x1011 | Logical Non-Volatile Device | 502.563 GiB | 0x0024    | DIMM9_CPU1
0x1021 | Logical Non-Volatile Device | 502.563 GiB | 0x0028    | DIMM7_CPU1
N/A    | DDR4                        | 64.000 GiB  | 0x000f    | DIMM3_CPU0
N/A    | DDR4                        | 64.000 GiB  | 0x0013    | DIMM1_CPU0
N/A    | DDR4                        | 64.000 GiB  | 0x0022    | DIMM3_CPU1
N/A    | DDR4                        | 64.000 GiB  | 0x0026    | DIMM1_CPU1

root@truenas[/home/admin]# ipmctl show -memoryresources
MemoryType   | DDR                 | PMemModule | Total
=======================================================================
Volatile     | 1004.000 GiB        | 0.000 GiB  | 1004.000 GiB
AppDirect    | -                   | 0.000 GiB  | 0.000 GiB
Cache        | 256.000 GiB         | -          | 256.000 GiB
Inaccessible | 17179868180.000 GiB | 0.000 GiB  | 17179868180.000 GiB
Physical     | 256.000 GiB         | 0.000 GiB  | 256.000 GiB

root@truenas[/home/admin]# ipmctl show -dimm
DimmID | Capacity  | LockState | HealthState  | FWVersion
===========================================================
0x0011 | 0.000 GiB | Unknown   | Unmanageable | N/A
0x0021 | 0.000 GiB | Unknown   | Unmanageable | N/A
0x1011 | 0.000 GiB | Unknown   | Unmanageable | N/A
0x1021 | 0.000 GiB | Unknown   | Unmanageable | N/A

root@truenas[/home/admin]# ipmctl show -socket
 SocketID | MappedMemoryLimit | TotalMappedMemory
==================================================
 0x0000   | 4608.000 GiB      | 502.000 GiB
 0x0001   | 4608.000 GiB      | 502.000 GiB
 
M

metricv

New Member
Jun 8, 2024
5
4
3
Update: I FINALLY GOT IT WORKING!!!!

Code: 
P: 512GB PMem 100 Module
D: 32GB DDR4-2666 RDIMM

+-----------------------------------------------+
|                      CPU0                     |
+-----------------------------------------------+
|          IMC1         |          IMC0         |
|  Ch5  |  Ch4. |  Ch3. |  Ch0. |  Ch1. |  Ch2  |
| 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
| P |   | D |   | D |   |   | D |   | D |   | P |

+-----------------------------------------------+
|                      CPU1                     |
+-----------------------------------------------+
|          IMC1         |          IMC0         |
|  Ch5  |  Ch4. |  Ch3. |  Ch0. |  Ch1. |  Ch2  |
| 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
| P |   | D |   | D |   |   | D |   | D |   | P |
This matches Dell technical books exactly, and it works. I also noticed that the POST time became much shorter compared to the original 1TB setup.

TL;DR: When playing with OEM stuff, try not to innovate and go by the books.
 
  • Like
Reactions: joerambo
You must log in or register to reply here.
Top Bottom