Supermicro H12SSL-i, H12SSL-C, H12SSL-NT, H12SSL-CT boards - notes, experiences

[lihp]

So far I have my fair share of experience with those boards overall and in-depth with the H12SSL-NT. Some notes are imho in order:

Overall

1. Those boards are fast. From my (our) tests they are among the fastest you can run EPYC on.
2. There were no compatibility issues under Windows or RHEL/CentOS/AlmaLinux. Fedora runs into weird issues starting from installation. Debian, Ubuntu installed fine too (apart from the current Ubuntu-installer issues with software RAID on boot drives).
3. I didnt encounter any stability issues with various hardware. Broadcom, Mellanox and friends all ran out of the box.

Hardware

1. The boards are standard compliant. Even though the compatibility list is short, there were no issues with different items not listed. Most notably M.2 NVME drives from various manufacturers and memory from Kingston (more about that later).
2. M.2 - it seems as if any drive compliant up to NVME 1.3c will work just fine (no wonder since its mostly the CPU, yet nonetheless they work).
3. Controller - Broadcom and Supermicro have a story together. So its no wonder I didnt see any issues with Broadcom RAID or HBA adapters.
4. Memory - Kingston offers a "product finder". I found it reliable for these boards. Different Kingston memory according their product finder just works.
5. H12SSL-C, H12SSL-CT: Both boards are almost identical and offer a Broadcom 3008 controller onboard. The T in the CT just declares that the Ethernet ports are 10G ports. Thats it, no magic.
6. H12SSL-i, H12-SSL-NT: Both boards offer slimline SAS ports (SFF-8654), one (1) in case of the "-i", two (2) in case of the "-NT". The slimline SAS ports can be used as SATA-ports (but 8i) or NVME ports (also 8i). The different usage can be set by BIOS or jumper. Each Slimline port is connect with x8 PCIe lanes to the CPU. In case of the "-i" there are additional SATA ports available compared to the "-NT".
7. Usage of the Slimline SAS ports require Supermicro specific cables (not expensive but sometimes hard to come by). For the Slimline SAS ports the options are direct connect for SATA/SAS/NVME (breakout cables) or backplane connect (due to the PCIe 4.0 8i nature of the port you need "uncommon" breakout cables for most backplanes too). If looking for cables, the most important part are the xi declaration and to make sure the endpoints xi sum up to 8i. Example: SATA/SAS breakout cables (usually without i since each is one lane) is Slimline SAS (SFF-8654) to 8x SATA/SAS. In case of NVME direct connect its a Slimline SAS (SFF-8654) 8i breakout cable to 2x 4i U.2/U.3 (SFF-8639).

OS:

1. Windows Server 2019 works fine.
2. RHEL 8.4/CentOS/AlmaLinux works fine. There are the usual log entries, which are easy to come by:
   1. error - watchdog: if you dont use the watchdog, just blacklist the sp5100_tco kernel module.
   2. error - mcelog: mcelog doesnt work on AMD CPUs, just stop and disable and use rasdaemon instead.
   3. If you use Gnome (you shouldnt on a server ), by then pulseaudio (part fo default graphical install) runs haywire as well as Bluetooth.
3. Fedora - installation requires manual intervention.
4. Ubuntu - works.
5. Debian - works.

Comments/Notes:

1. SATA cables for the slimline SAS ports are angled towards the CPU. Be cautious since you can easily break the cables this way and they are always bent during operations.
2. Use heatsinks for the M.2 drives. Even with great case air flow, high end M.2 drives will get close to critical temperatures under heavy sustained load. I suggest "be quiet! MC1 Pro, M.2 SSD-heatsink" - they are a tight fit, loosen both drives, insert and gently press down both together. Under heavy M.2 load they went up to 85°C (=critical) without heatsink. With heatsink temperatures dropped to 66°C max (10+ hours sustained load). Meaning without heatsink your M.2 drives may bail out under heavy load. With heatsink your M.2 drives never even come close to critical temperature, even under the heaviest sustained load.
3. The IPMI password is under the first M.2 slot and on the transparent throwaway CPU cover (You can easily miss the password if you dont know). Alternatively you can reset it to ADMIN/ADMIN (jumper) or change it from command line with ipmi tools.

 

[uldise]

I suggest "be quiet! MC1 Pro, M.2 SSD-heatsink" - they are a tight fit, loosen both drives, insert and gently press down both together.

so you need two of them, one for each M.2?
how you attach them? are there screws?

i have H12SSL-C, runs with Proxmox just fine. i have one M.2 drive attached, and are planning to add second one. Now running without a heatsinks.

 

[lihp]

so you need two of them, one for each M.2?
how you attach them? are there screws?

Its just a metal casing with screws and self-adhesive heatpads.

 

[lihp]

i have one M.2 drive attached, and are planning to add second one. Now running without a heatsinks.

If you have it for personal use with a burst use here and there, you are all good. If you have sustained burst loads, you will need a heatsink.

 

[uldise]

Its just a metal casing with screws

where you screw them? H12SLL comes with some plastic piece to attach M.2..

 

[lihp]

where you screw them? H12SLL comes with some plastic piece to attach M.2..

 

[lihp]

Like I said, its a tight fit. You need to loosen both M.2 drives, make sure the heatsink screws are as deep in as possible without breaking them. And then push down both at once gently and close the plastic piece.

PS - edit: its really a tight fit. I cant stress it enough - make both drives loose and push both gently together down once in heatsink, without breaking anything. Its like 0.5 mm space lacking and thats what you have to overcome. Tight fit - but works.

 

[ca3y6]

I ran into some stability issues with the TPM AOM-TPM-9665V and the H12SSL-i (blue screens, followed by the BIOS freezing during the reboot process). Supermicro support suggested I replace the TPM module, waiting for the part right now. Without the module the server is stable. Has anyone experienced anything similar?

 

[vincococka]

Thanks a lot for great summary.
I'm currently waiting for TPM 2.0 AOM-TPM-9665V for H12SSL-i - which was approved by SuperMicro support.

I've installed DDR4 Micron 3200MHz SR 16GB * 8pcs - everything seems stable so far.
OS Windows 10 2004 + June 2021 update + latest SP3 drivers from SuperMicro site
Cooler is Noctua NH-U9 TR4-SP3 - but next time I'll be considering SuperMicro cooler/heatsink as Noctua exhausts warm air to the top,
while SM coolers exhausts warm air to the back of the case (ATX CASE in my case ).

Idle consumption is cca 40-45W with Epyc 7313p 16c 3.0GHz (SMT OFF) - measured by HWInfo / CoreTemp - quiet a lot, but yeah - who wants server CPU in a workstation than needs to respect this idle consumption.

While room temp is now cca 28-30 deg C - hot summer here in Europe:
Idle TEMP: 45-47 deg C
Load Temp: 60-65 deg C (prime95)

Observation so far:
- IPMI 2.0 looks like a beta product - at least it crashed couple of times and I had to perform IPMI reset
- design of IPMI 2.0 reminds me of 2005 year of web 2.0 (personal opinion)
- IPMI Active Directory integration requires license
- BIOS updates via IPMI are now included/available without license - previously this feature has been licensed.
- BIOS behaves buggy while enabling/disabling NIC1 EFI/Firmware support
- SATA disks are detected/discovered before NVMe drives during Windows 10/2019 setup -> Windows installer will try to create UEFI boot partition on first drive in the system
- BIOS does not have screen which will show detailed information about installed NVMe drives and slot occupacy.

 

[lihp]

I'm currently waiting for TPM 2.0 AOM-TPM-9665V for H12SSL-i - which was approved by SuperMicro support.

Same here - ordered, waiting. So I didn't answer so far since I lack hands-on experience.

- IPMI 2.0 looks like a beta product - at least it crashed couple of times and I had to perform IPMI reset
- design of IPMI 2.0 reminds me of 2005 year of web 2.0 (personal opinion)

I experienced it quite the opposite, not only with these boards, but any Supermicro board. Yet I agree it always looks outdated - actually fine with me, since I expect changes on core functionalities to be handled very conservatively. I actually wish others would do the same and go function and reliability over looks.

- IPMI Active Directory integration requires license

Actually I consider it a gimmick. The IPMI interface is normally a separate network.

- BIOS behaves buggy while enabling/disabling NIC1 EFI/Firmware support

Never experienced it. I suggest a call with Supermicro support. Especially JW and HP from Supermicro support are competent and helpful and react fast on anything buggy.

On the last two points nothing I can add since I never tried. On Windows 10 Id go simply with an unattended setup or alternatively just disable all other drives except boot drive. After all thats a windows issue. Afaik you also need Windows 10 Enterprise for Epyc CPUs.

 

[uldise]

Hi, i'm planning to add AOC-SLG3-4E4T | Add-on Cards | Accessories | Products - Super Micro Computer, Inc. to my board and connect it with direct attach Oculink to U.2 cables my NVME drives. i have a spare 5.25 slot available in my case for that.
have anyone tried this combination?
Thanks

 

[i386]

Afaik you also need Windows 10 Enterprise for Epyc CPUs.

"It depends"
For s single socket and up to 128GB windows 10 home edition can be used [1]. For dual socket or 128GB - 2TB ram systems you will need at least the professional edition [2] and for 4 socket and 2+ TB ram system you will need the professional for workstation edition

[1]

Memory Limits for Windows and Windows Server Releases - Win32 apps

This topic describes the memory limits for supported Windows and Windows Server releases.

docs.microsoft.com

[2] Windows 10 versions CPU limits

 

[lihp]

Hi, i'm planning to add AOC-SLG3-4E4T | Add-on Cards | Accessories | Products - Super Micro Computer, Inc. to my board and connect it with direct attach Oculink to U.2 cables my NVME drives. i have a spare 5.25 slot available in my case for that.
have anyone tried this combination?
Thanks

No didnt try, but: the AOC card is PCIe Gen3, so any PCIe 4 NVME drive would run at reduced speed. May not make much of a difference in performance, but I didnt test that. Overall it should work anyways.

For PCIe 4.0 drives a statement directly from Broadcom support:
I think that the P41132p or the 9500-16i would work fine.

The P41132p is native NVMe and are not treated as SCSI devices so applications such as VROC and other native NVMe tools will work as expected.

The 9500-16i uses the SCSI layer and most tools do not see the drives as a native NVMe drive.

The other advantage to the P411 is that it supports x1, 4, and x8 link.

Here you have to look closely at the cables, since standard Broadcom cables for the 9500-16 only allow x1 (1i), but with different cables you can go 4i for each drive (which is possible with Broadcom:
9500-16i would give you the maximum bandwidth indeed.

The problem becomes connectivity.
If you direct connect, the only cable we have is

05-60006-00

1m

x8 SFF-8654

Eight U.3 SFF-8639

The cable only allows x1 lane.
The 05-50065-00 is x4, but for the 9460-16i.
You may be able to find x4 SFF-8654 like this:

Slim SAS SFF-8654 4i Straight to SFF-8639 U.2 Cable

We have not tested any other cables, but they should work.

 

[uldise]

i'm fine with PCIe Gen3 speeds. Gen4 stuff is so new and are more expensive - controller, cables, drives..
and that's why i'm asking - will that Gen3 stuff work just fine with H12SSL?

 

[ejbs]

I am working on a build with a 7443P and an H12SSL-i. I’m still waiting for my 7443P to be shipped (hopefully by Monday). I was using the IPMI web interface of the H12SSL-i but was not able to get the case fans to spin. Is that expected? Does it require the CPU to be installed before it can spin up the case fans?

 

[uldise]

Yes, fans start spinning when you power-on. without CPU you can't power-on..

 

[lihp]

i'm fine with PCIe Gen3 speeds. Gen4 stuff is so new and are more expensive - controller, cables, drives..
and that's why i'm asking - will that Gen3 stuff work just fine with H12SSL?

From me it's a definite yes. It's a PCIe card for Supermicro boards, so the card itself should work anyways. The drives are either handled by the card, so they work that way. Or the card acts as HBA, in which case there would only be an NVME protocol downgrade, which in return will work due to downward compatibility.

Supermicro support would say: the AOC is not certified for H12SSL-xx, only H11SSL-xx. We advise against it.

 

[uldise]

Thanks!
and what about a cables? i just hope this one should work: Supermicro 55cm OCuLink to U.2 PCIE with Power Cable (CBL-SAST-0956)

but looks like i should wait for availability of this card here in EU, can't find it anywhere..

 

[lihp]

Thanks!
and what about a cables?

Just verify with Supermicro support, they are super helpful. Ideally just go to their e-store, when support is on and catch them via the chat.

 

[lihp]

but looks like i should wait for availability of this card here in EU, can't find it anywhere..

You just need to find a Supermicro partner or a company specialized in Supermicro. They usually have to order and get the stuff within 1-2 weeks.

Still here is what I would do in your case:

1. Read the entry notes by me about cards and cables.
2. Alternate 1: Buy a used Broadcom P411W-32P HBA (Linux). Here in Germany you get one on ebay for 535,00 €.
3. Alternate 2: If you are in for risk, buy a PCIe 4.0 bifurcation card like Delock Produkte 89030 Delock PCI Express x16 Karte zu 4 x intern SFF-8654 4i NVMe - Bifurcation - they should work, but I dont know for sure.
4. Get cables and make sure they are suited for PCIe 4.0 and for 4 lanes for either alternative.
5. Direct connect your PCIe 4.0 NVME drives - one for each slot.