EPYC 7002/7003 MB choice

[KaneTW]

I'm building a single-socket EPYC server. My original plan was to use H12SSL-NT, but availability is worse than I expected. I plan to have up to 8NVMe SSDs, so having 2xSlimSAS x8 PCIe gen4 is very useful. A full set of PCIe gen 4 slots is a must.

Potential alternatives include Asrock ROMED8-2T and Tyan S8030. S8030 has 2x SlimSAS x8, but worse availability/price in Europe than the Asrock. Asrock only has 2x Oculink x4 and is more expensive than the H12SSL-NT, but is available.

The server will be running ESXi 8, so BCM NIC isn't an issue.

Any advice/experiences with EPYC MBs that could help me decide?

 

[RolloZ170]

Gigabyte MZ32-AR0 not an option ?

 

[KaneTW]

Gigabyte MZ32-AR0 not an option ?

No PCIe gen 4 for the majority of slots.

 

[KaneTW]

only slot1 and 2 ( behind a switch )

The SlimSAS are gen3, which is not something I want.

 

[gb00s]

No PCIe gen 4 for the majority of slots.

?

 

[KaneTW]

?

OCP, M.2, two expansion slots, and most importantly all SlimSAS ports are only gen3.

 

[gb00s]

The SlimSAS are gen3, which is not something I want.

Not entirely correct as SlimSAS on the rev3.0 version is Gen4

 

[KaneTW]

Ah, thanks. Ok, that makes it a possible consideration.

 

[gb00s]

Also rev1.0 SlimSAS just shares with Slot7 Gen4. So if you dont use SlimSAS you still get full X16 Gen4. That makes it just 3 slots PCIe Gen3 (x16 + x8 with bi-furcation option plus OCP Gen3). But you have almost 10 slots available ... I mean which 7002/7003 board offers this? And its cheap and cost almost 200US$ now. You can prep it up for Milan etc ...

 

[KaneTW]

Hmm, ok. I'll take a look. I don't see it cheap outside of China exports but I'm already importing a China EPYC CPU so why not.

 

[KaneTW]

I really don't like the slot arrangement on the Gigabyte board. Tyan S8030 is about equally expensive from China right now btw.

 

[Kizune]

How about Tyan S8030 series? Supports 2nd/3rd generation of Epyc, has two SlimSAS x 8 and 5 full size PCIe Gen4 x16 slots. Can be found for a reasonable price.

AMD single-socket motherboards | TYAN Computer

Tyan's AMD single-socket motherboards support the latest AMD EPYC processors for HPC, AI, deep learning, datacenter, storage applications.

www.tyan.com

 

[KaneTW]

Yep, this is one of the strongest contenders.

S8030: $300 used from CN; good docs, lowest free slot count (2x16 dual-width) after 8xNVMe, 40/100G NIC, HBA. Unclear what the differences between the two versions
MOA and OOY are (seems like MOA is gen4 M.2 and some "pin definitions modified" on the NVMe SlimSAS, plus additional fan headers)

MZ32: not suitable for GPU pcie4.0 use without risers, and risers make everything too complicated

H12SSL-NT: highest free slot count with config as above (5x16), but unavailable or very expensive, $700 new now or $560 mid-Sep

H12SSL-i: one less slot than -N, $500 new, requires a $120-200 retimer to get to same NVMe count as -NT so same price in the end

ROMED8-2T: 3x16 left after config, same price as -NT, but available. too expensive after retimer. Overall uninteresting.


I don't plan to install a GPU *yet*, but one of the servers it's replacing is a GPU server so I need to keep that option open.

I think ultimately my decision is between $300 for a used S8030 and $700 for a new H12SSL-NT.

 

[adman_c]

Yep, this is one of the strongest contenders.

S8030: $300 used from CN; good docs, lowest free slot count (2x16 dual-width) after 8xNVMe, 40/100G NIC, HBA. Unclear what the differences between the two versions
MOA and OOY are (seems like MOA is gen4 M.2 and some "pin definitions modified" on the NVMe SlimSAS, plus additional fan headers)

MZ32: not suitable for GPU pcie4.0 use without risers, and risers make everything too complicated

H12SSL-NT: highest free slot count with config as above (5x16), but unavailable or very expensive, $700 new now or $560 mid-Sep

H12SSL-i: one less slot than -N, $500 new, requires a $120-200 retimer to get to same NVMe count as -NT so same price in the end

ROMED8-2T: 3x16 left after config, same price as -NT, but available. too expensive after retimer. Overall uninteresting.


I don't plan to install a GPU *yet*, but one of the servers it's replacing is a GPU server so I need to keep that option open.

I think ultimately my decision is between $300 for a used S8030 and $700 for a new H12SSL-NT.

If you don't need SAS support, most of these boards have a ton of SATA ports built in so that could save you a PCIE slot. Obviously if you need SAS, you need a HBA.

 

[KaneTW]

If you don't need SAS support, most of these boards have a ton of SATA ports built in so that could save you a PCIE slot. Obviously if you need SAS, you need a HBA.

The case I'm using has an expander backplane, so HBA is required.

 

[KaneTW]

Ok, this thing is driving me nuts in my research: NVMe hotplug compatibility.

From what I understand, there's a SMbus header on my NVMe backplane and some of the motherboards for NVMe hotplug support, e.g.

S8030: J44 (page 21) https://i.kane.cx/sfdtPL
MZ32-AR0: BP_1 (page 24) https://i.kane.cx/QsuXtF
ROMED8-2T: CPU1_HSBP1 (page 39) https://i.kane.cx/2ouX9m

Supermicro H12SSL-NT doesn't have it or anything like it.

Is it required for hot-plug support to work, or is it some unused thing? Does the H12SSL-NT support NVMe hotplug at all? There's 0 documentation on it.

I sent an email to Supermicro/Tyan support as well because NVMe hot plug seems to be some severely underdocumented dark magic.

E: Supermicro "Most of our motherboards have a specific I2c connector on the board for NVMe, which is required for this [hot-swap] to work, but this one does not." (hot-plug is unsupported, only hot-swap is supported)

 

[KaneTW]

I went ahead with the H12SSL-NT regardless and yeah, hot swap doesn't work at all (with ESXi). Doesn't even detect an unplugged PCIe device.

 

[drdepasquale]

NVMe hot plug compatibility is iffy at best. Supermicro H12 boards do have hot plug functionality to some degree. I have a workstation built with Supermicro H12DSi-NT6. The NVMe drives plugged into the slimline SAS ports are hot-pluggable. Windows even shows the option to eject the drives as if they were removable storage. It functions the same as SATA hot plugging. The NVMe drive installed in the m.2 slot is not hot-pluggable at the BIOS level or the operating system. The single CPU version of an H12 board (H12SSL-NT and others) is likely very similar in functionality.

 

[KaneTW]

Interesting. I tried with ESXi 8 and it doesn't detect them as hotpluggable at all.

What chassis/backplane are you using?

 

[drdepasquale]

Interesting. I tried with ESXi 8 and it doesn't detect them as hotpluggable at all.

What chassis/backplane are you using?

I'm not using a Supermicro chassis or a backplane. I am using a Corsair 900D tower with the NVMe drives attached directly to the motherboard using a SFF-8654 to U.2 SFF-8639 cable.