NVDimm support on AMD Epyc

Rand__

Well-Known Member
Mar 6, 2014
5,381
1,147
113
Visually inspected Boards/Bios:
  • SM H11SSL - no sign of NVDimm settings
  • tyan TS75-B8252 - some settings, quite sparse
  • tyan s8030 - some settings, quite sparse
  • asrock rack romed6 - no sign of NVDimm settings
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hi,

based on https://forums.servethehome.com/index.php?threads/intel-900p-280gb-u-2-drives-229.32585/ I wondered whether Epyc boxes would support NVDimm-Ns (not the Intel NVDimm-P = Optane ones), -N predates Optane but has not been widely adopted (despite its memory like performance which makes it an excellent choice for cache drives/slog). Today its not playing a big part any more, most providers seem to have ceased development in favor of new CXL based solutions.

NVdimms are a rather complicated system, they need mainboard support + OS support (and for ultimate performance optimized applications, but even basic block performance is great as mentioned). O/c given their age they only run up to 2666MHz to not an ideal part for Epyc, but matched for the first gen Scalable boxes which were their main domain.

However these are relatively cheap to come by, are blazingly fast so potentially interesting for the advanced user. They run fine in FreeBSD/Linux/Windows so are usable for solutions like TNC/ZoL/S2D (not ESXi unfortunately unless you go branded HPE/Dell which have working options there).

So long intro, just to give you some background why I ask;)

I am looking for someone with an (Supermicro or other) Epyc system being able to do a quick boot and check if there are option like these from my X11SPH-nCTPF present. If so then this might work, if not then probably not. O/c SM did not do a very thorough implementation of these as they did not see many requests for NVDimm-Ns and thus probably did not spend much development resources on this.

1618645929955.png


@i386 already confirmed that NVDimms-N are theoretically supported here https://forums.servethehome.com/index.php?threads/intel-900p-280gb-u-2-drives-229.32585/post-300985, but whether its actually implemented on the board ...
1618646125211.png


O/c I am happy to get multiple boards checked if more than one person is willing to, so please provide board and status so if we find some which might work we can create a list.

Thanks a lot.
 
Last edited:

RageBone

Active Member
Jul 11, 2017
363
97
28
SM H11SSL with a BIOS version of 2.0a and a 2S1705... Rome ES, no dice with my NVDIMM-N that i got from you, no power gem though.

With just the Nvdimm, it got stuck at D0 which seems to be a memory training error.
This specific CPU has a few dead channels but behaves identical in all channels, with just the Nvdimm.
A good stick of mem also posts D0 in a dead channel.

In addition with a good stick of memory and in confirmed working channels, the system posts and reports both sticks as existing and supposedly working.
If the 24GB of mem and SPD info can be believed.

The Nvdimm initially on power-on lights up all three LEDs with the Board taking its time at postcode 3F until the LEDs blink a few times and turn of, only the green one remaining lit on the stick, POST then proceeds to either get stuck at D0 or continue.

Postcode 10 seems to be "no Mem Installed / found" and code 15 seems to be the opposite.

@Rand__ tag after edit for notification
 

Attachments

Last edited:
  • Like
Reactions: Rand__

Rand__

Well-Known Member
Mar 6, 2014
5,381
1,147
113
Thanks a lot.
Usually it should not matter if you have attached a PowerGem or not, its just not persistent then (at least I never noticed different behavior)

Unless you skipped specific Bios pages there is nothing there indicating NVDimm-N support at all unfortunately.

Now different Boards have different support teams at Supermicro, so theoretically its possible another board supports them, but at this point I dont expect it (given how badly they have been supported on their primary platform;
although there used to be tested NVDimms for this board (Naples, EOL'ed by now, MEM-DR416L-CL01-NV26)
 
Last edited:

RageBone

Active Member
Jul 11, 2017
363
97
28
i could be overlooking something, and it is an early ES Bios, hence not that trustworthy.
gona try the truenasScale on it and see if that sees it as nvdimm when i add the tunable.
 

Rand__

Well-Known Member
Mar 6, 2014
5,381
1,147
113
not sure re scale but on regular TNC the tunable is not needed any more
 

jpmomo

Active Member
Aug 12, 2018
176
55
28
I was able to check a couple of tyan mb with the following results:

tyan TS75-B8252

1618726684176.png

tyan s8030

1618727580617.png


I will check on a couple of asrock rack mb tomorrow (romed6 and romed8). Unfortunately my supermicro h12ssl is being rma'd (for the second time!) and probably won't have that back for a while. I have another sm h12ssw that I can try and piece together to check on any available config for nvdimms in the bios.
 
  • Like
Reactions: Rand__

mirrormax

Member
Apr 10, 2020
92
38
18
gigabyte rome board has the same option not sure if its just a generic amd bios option or actually supported by the board, no mention of it on gigabyte site.
 

RageBone

Active Member
Jul 11, 2017
363
97
28
had a look at the SM H11ssl Naples Part of the bios, no mentions of nvdimm features.
What should look for in Truenas to see if it works?

sysctl -a | grep
nvdimm
pmem

both give some results.
 

jpmomo

Active Member
Aug 12, 2018
176
55
28
no reference to nvdimm on asrock rack romed6. Still need to check on romed8 and a sm h12.
 

RageBone

Active Member
Jul 11, 2017
363
97
28
booted up a manjaro on my test machine for another project with the nvdimm still in place.
manjaroDmesg.png