512 GB Intel Optane DDR4-2666 RDIMM PMEM NMA1XBD512GQS

[jason879]

I'm a stock market investor and speculator at night and from that point of view and correct me if I am wrong but those 512GB PMEM 100 just had a RRP of 2000 USD per piece. Now 120 or 130. I think they won't come down more, these are imho EOL and firesale prices. Next stop might be Unobtainium City.
With Reddit imploding a little more every day these days and users jumping onto lemmy or here or level1, maybe it'd be a good time to rescue the r/homelab slogan. ;-) Don't forget you probably want an -L suffix CPU for that.

I was waiting for 8200 and mobo to come down. Recently I got a deal on the mobo @$250 so I could start to build my large memory rig.

I'm planning to get 8259CL for ~$275 or less and a WD SN640 7.68T ~$300. So the total will be round $1K usd. For PMEM I'll see how that works then decide if I will get more, no rush to get it fully loaded ATM. These parts are always going down in most cases.


@RolloZ170 thanks for the info. I was reading all your threads regarding mod TDC=255A on different platforms(Dell 7820/R740/X11-SPA). I end up getting EPC621D8A which is mentioned on one of you reply. It supports special SKU out of the box along with few SuperMicros.

 

[RolloZ170]

correct me if I am wrong but those 512GB PMEM 100 just had a RRP of 2000 USD per piece. Now 120 or 130.

2000usd new with warranty. 120 is too much if they die after 4 weeks of use.

 

[Stephan]

120 is too much if they die after 4 weeks of use.

They might and I know full well that it's a gamble. So I bought 7 with 6 in use and 1 as spare. I even calculated 50:50 they wouldn't even show up in the Asrock and Supermicro boards with the 82xx CPUs I have. Funny enough everything works. That seller appears to sell new in box modules even. Datasheet says 300 PBW. In the cluster I might write 1 TB per day per module. Would take 822 years to wear them out.

 

[jason879]

Got two sticks today nicely packed in bubble envelop via Fedex. Now waiting for other parts.

 

[jasonsansone]

I waited too long on this and now I can’t find anything even close in price

 

[jason879]

I got everything setup today. In memory mode the speed is not bad(quad channel). Doubled than my EPYC 7302P esxi setup (maybe it's not fine tuned). I've would bought 2 more to fill up the slots... Anyways it's good enough.

My setup as below for anyone whom is interested to build in the future. It works out of the box no mod needed.(Only 2 jumpers to endable NVDIMM support)

Intel 8259CL x 1

ASRock EPC621D8A x 1

Intel Optane 512GB x 2

Hynix 16GB 2666V x 4

WD SN640 7.68T x 1

CPU Cooler x 1

 

[gb00s]

That's a 1:16 ratio of DRAM vs PMEM. I can feel from here the Optanes are sad.

 

[mrpasc]

Want to learn and understand, so why? Intel says 1:4 to 1:16 is okay?

Frequently Asked Questions for Intel® Optane™ Persistent...

Answers questions about Intel® Optane™ Persistent Memory.

www.intel.com

 

[nasbdh9]

The ratio of DDR memory to DCPMM depends on your use case
DCPMM in memory mode is more like a hardware SWAP
According to 32*6+512*6, the full-speed memory you have is 32*6, and the part exceeding 32*6 will be written into DCPMM
Assuming your active memory portion will never exceed 32*6, why configure more DDR memory?

 

[jason879]

Got 2 more sticks and the order has to be set like the following. (Not interleaved DRAM | PMEM | DRAM | PMEM)

1:16 DRAM to PMEM ratio seems like a recommendation not a hard requirement. ESXI 8.0 is able to recognize 2TB (64GB dram cache) without issue.

 

[jason879]

I waited too long on this and now I can’t find anything even close in price

Another seller is selling similar item NMA1XXD512GPS @$172.99 USD now. They accepted my offer @$160 USD
I guess GPS is the production version, isn't it? Stephan

 

[NablaSquaredG]

If anyone wants to get rid of their 512G Optane DCPMM DIMMs, let me know

 

[jason879]

If anyone wants to get rid of their 512G Optane DCPMM DIMMs, let me know

This seller still have many. Intel NMA1XXD512GPS 512GB Optane Persistent Server Memory PC4 RAM | eBay

 

[NablaSquaredG]

This seller still have many. Intel NMA1XXD512GPS 512GB Optane Persistent Server Memory PC4 RAM | eBay

yes but nearly 40% more expensive

 

[chrgrose]

Is there a test to show how worn out Optane dimms are? Or do you just use them until they die?

 

[NablaSquaredG]

Is there a test to show how worn out Optane dimms are? Or do you just use them until they die?

You can check it in the BIOS at least (probably also via CLI)

https://www.intel.com/content/dam/support/us/en/documents/memory-and-storage/data-center-persistent-mem/Intel-Optane-DC-Persistent-Memory-BIOS-settings.pdf

See Slide 13 ("Percentage Remaining")

 

[Stephan]

Or use ipmctl on Arch. Here are some examples for reference:

# Configuration

ipmctl show -topology
ipmctl show -dimm
ipmctl show -a -dimm
ipmctl show -memoryresources
ipmctl show -goal
ipmctl show -region

# Firmware

ipmctl show -firmware
ipmctl show -dimm -firmware
ipmctl show -d ARSStatus -dimm

# Health

ipmctl show -d HealthState,HealthStateReason -dimm
ipmctl show -a -sensor
ipmctl show -a -sensor MediaTemperature
ipmctl show -a -sensor ControllerTemperature

# Set Goal

ipmctl create -goal PersistentMemoryType=AppDirectNotInterleaved
ipmctl create -dimm 0x0101 -goal PersistentMemoryType=AppDirectNotInterleaved

# Create Namespace

ndctl create-namespace --mode sector
ndctl list --regions --human
ndctl list --human

Also good luck trying to wear out these Optanes. They'll just last until you are fed up with Cascade Lake and get something better.

 

[Stephan]

Disagree @NablaSquaredG?

 

[NablaSquaredG]

Disagree @NablaSquaredG?

no

Also good luck trying to wear out these Optanes. They'll just last until you are fed up with Cascade Lake and get something better.

I think that's a funny statement

 

[jasonsansone]

So I tested throughput and IOPS for ext4 and ZFS w/ and w/o LZ4 to get some ballpark figures.

Hardware: Supermicro X11SPL-F BIOS 3.9 CPU 8259CL RAM 1x32 GB Samsung RDIMM 2666 PMEM 1x512 GB Optane Firmware 01.02.00.5446. Both memory modules on first IMC and first channel.

Linux kernel 6.2.16 booted from USB with options

root=PARTUUID=af8fa598-02 rw console=ttyS0,115200n8 console=tty msr.allow_writes=on rootflags=compress=zstd intel_iommu=on audit=0 delayacct mitigations=off init_on_alloc=0 init_on_free=0

Optane Setup

ipmctl create -goal PersistentMemoryType=AppDirectNotInterleaved
ndctl create-namespace --mode sector

ext4 Setup

mkfs.ext4 -m0 -Elazy_itable_init=0 /dev/pmem0s

ZFS Setup

zpool create -f -o ashift=12 -o cachefile=/etc/zfs/zpool.cache -O dnodesize=legacy -O normalization=formD -O mountpoint=none -O canmount=off -O compression=lz4 -O aclinherit=passthrough -O acltype=posixacl -O xattr=sa -O relatime=on -O dedup=off zpool /dev/pmem0s

zfs create zpool/comp
zfs create zpool/nocomp

zfs set compression=lz4 mountpoint=/mnt/ssd/comp zpool/comp
zfs set compression=off mountpoint=/mnt/ssd/nocomp zpool/nocomp

All numbers from a tweaked fio script called cdm_fio.sh approximating "CrystalDiskMark".

ZFS no compression:
View attachment 29385

ZFS with LZ4 compression:
View attachment 29383

ext4:
View attachment 29384

Your DCPMM’s are in sector mode. Did you ever compare ZFS to EXT4 using fsdax?