NVDimm support on AMD Epyc

[Rand__]

Visually inspected Boards/Bios:

• SM H11SSL - no sign of NVDimm settings
• tyan TS75-B8252 - some settings, quite sparse
• tyan s8030 - some settings, quite sparse
• asrock rack romed6 - no sign of NVDimm settings

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hi,

based on https://forums.servethehome.com/index.php?threads/intel-900p-280gb-u-2-drives-229.32585/ I wondered whether Epyc boxes would support NVDimm-Ns (not the Intel NVDimm-P = Optane ones), -N predates Optane but has not been widely adopted (despite its memory like performance which makes it an excellent choice for cache drives/slog). Today its not playing a big part any more, most providers seem to have ceased development in favor of new CXL based solutions.

NVdimms are a rather complicated system, they need mainboard support + OS support (and for ultimate performance optimized applications, but even basic block performance is great as mentioned). O/c given their age they only run up to 2666MHz to not an ideal part for Epyc, but matched for the first gen Scalable boxes which were their main domain.

However these are relatively cheap to come by, are blazingly fast so potentially interesting for the advanced user. They run fine in FreeBSD/Linux/Windows so are usable for solutions like TNC/ZoL/S2D (not ESXi unfortunately unless you go branded HPE/Dell which have working options there).

So long intro, just to give you some background why I ask

I am looking for someone with an (Supermicro or other) Epyc system being able to do a quick boot and check if there are option like these from my X11SPH-nCTPF present. If so then this might work, if not then probably not. O/c SM did not do a very thorough implementation of these as they did not see many requests for NVDimm-Ns and thus probably did not spend much development resources on this.




@i386 already confirmed that NVDimms-N are theoretically supported here https://forums.servethehome.com/index.php?threads/intel-900p-280gb-u-2-drives-229.32585/post-300985, but whether its actually implemented on the board ...



O/c I am happy to get multiple boards checked if more than one person is willing to, so please provide board and status so if we find some which might work we can create a list.

Thanks a lot.

 

[RageBone]

SM H11SSL with a BIOS version of 2.0a and a 2S1705... Rome ES, no dice with my NVDIMM-N that i got from you, no power gem though.

With just the Nvdimm, it got stuck at D0 which seems to be a memory training error.
This specific CPU has a few dead channels but behaves identical in all channels, with just the Nvdimm.
A good stick of mem also posts D0 in a dead channel.

In addition with a good stick of memory and in confirmed working channels, the system posts and reports both sticks as existing and supposedly working.
If the 24GB of mem and SPD info can be believed.

The Nvdimm initially on power-on lights up all three LEDs with the Board taking its time at postcode 3F until the LEDs blink a few times and turn of, only the green one remaining lit on the stick, POST then proceeds to either get stuck at D0 or continue.

Postcode 10 seems to be "no Mem Installed / found" and code 15 seems to be the opposite.

@Rand__ tag after edit for notification

 

[Rand__]

Thanks a lot.
Usually it should not matter if you have attached a PowerGem or not, its just not persistent then (at least I never noticed different behavior)

Unless you skipped specific Bios pages there is nothing there indicating NVDimm-N support at all unfortunately.

Now different Boards have different support teams at Supermicro, so theoretically its possible another board supports them, but at this point I dont expect it (given how badly they have been supported on their primary platform;
although there used to be tested NVDimms for this board (Naples, EOL'ed by now, MEM-DR416L-CL01-NV26)

 

[RageBone]

i could be overlooking something, and it is an early ES Bios, hence not that trustworthy.
gona try the truenasScale on it and see if that sees it as nvdimm when i add the tunable.

 

[Rand__]

not sure re scale but on regular TNC the tunable is not needed any more

 

[jpmomo]

I was able to check a couple of tyan mb with the following results:

tyan TS75-B8252



tyan s8030




I will check on a couple of asrock rack mb tomorrow (romed6 and romed8). Unfortunately my supermicro h12ssl is being rma'd (for the second time!) and probably won't have that back for a while. I have another sm h12ssw that I can try and piece together to check on any available config for nvdimms in the bios.

 

[Rand__]

That looks promising, thank you very much

 

[mirrormax]

gigabyte rome board has the same option not sure if its just a generic amd bios option or actually supported by the board, no mention of it on gigabyte site.

 

[Rand__]

Hm I see. Maybe needs an actual test...

 

[RageBone]

had a look at the SM H11ssl Naples Part of the bios, no mentions of nvdimm features.
What should look for in Truenas to see if it works?

sysctl -a | grep
nvdimm
pmem

both give some results.

 

[Rand__]

A new 16g disk drive you can attach to a pool

 

[jpmomo]

no reference to nvdimm on asrock rack romed6. Still need to check on romed8 and a sm h12.

 

[Rand__]

Thanks

 

[RageBone]

booted up a manjaro on my test machine for another project with the nvdimm still in place.

 

[Rand__]

Thanks though

 

[nasbdh9]

Someone sells some nvdimm modules with 2933 frequency and original firmware 2.5 on the second-hand trading platform in China, but unfortunately I don’t have a platform that supports nvdimm at the moment.

 

[Rand__]

Yeah, these seem to pop up now when ppl move to 3rd Gen Scalable... no idea what powergem they need though and whether 2.50 is fine or needs updates...
How much are they in China?

 

[nasbdh9]

Yeah, these seem to pop up now when ppl move to 3rd Gen Scalable... no idea what powergem they need though and whether 2.50 is fine or needs updates...
How much are they in China?

Powergem should be universal. I don’t know how much inventory there is. Generally sellers don’t have the conditions to test these. A unit only costs about $100.

 

[Rand__]

I dont know re Powergems - I have been told that these are not interexchangeable (between different series), but I never gave it a try to be honest.

 

[nasbdh9]

5 new items by Chitose Ikeda

photos.app.goo.gl

Some dismantling