Supermicro H12SSL-i, H12SSL-C, H12SSL-NT, H12SSL-CT boards - notes, experiences

[zachj]

I sit next to an h12ssl-nt all day every day. It’s got an nvidia A10 and 16xSATA SSDs. The only thing that’s loud is the halfassed blower cooler I mounted on the GPU and the 80mm fan on the drive cage (icydock).

power consumption is definitely higher than a reasonable person can justify but I have electric baseboard heat so I figure it doesn’t really matter in the winter; my desktop works hard so my radiators don’t have to

 

[Pri]

Thought I'd update the thread on my H12SSL-NT. I got this board in June 2022 and I built a server using it. I've almost entirely maxed this board out and I'm not experiencing any glitches or weird behaviour even with very seldom combined hardware.

CPU I'm using is a 7773X Milan-X chip which has 64 cores, 128 threads and 768MB of L3 cache, this is for my workload anyway the ultimate chip for this motherboard with its enourmous amounts of cache thanks to the 3D V-Cache. Works wonderfully in this board even with its very high energy consumption when maxed out the VRM's on the board don't have any problems.

RAM wise as I posted on an earlier page I'm using 8 x 128GB Samsung 3200MHz ECC LR-DIMM's for 1TB of total capacity. I am running these all at their rated 3200MHz frequency & CL22 latency without issue on the 7773X and previously I ran these on a 7443p 24-core on the same board also without issue.

I'm using 7 PCIe cards including:
Intel X710-DA4 network card (4 port 10Gb SFP+)
Intel X550-T2 network card (2 port 10Gb RJ45)
Asus HyperM.2 card with 4 x 2TB 980 Pro SSD's (PCIe4 passive card in x4x4x4x4 bifurcation)
Asus HyperM.2 card with 2 x 2TB SN850 SSD's (Again in x4x4x4x4 bifurcation)
GLOTRENDS PU41 PCIe to U.3 card with 4 x U.3 Kioxia 7.68TB Drives (PCIe 4 passive card in x4x4x4x4 bifurcation)
NVIDIA RTX A4000 16GB Ampere graphics card
Broadcom 9500-8i Host Bus Adapter

In addition to this I've made use of the two on-board SlimSAS connectors for 4 x U.3 PCIe 4.0 Kioxia 7.68TB drives. I used HighPoint cables for this as I figured they would be reliable and they have been, no signal issues at all and the full PCIe 4.0 bus speed on all four drives at once works reliably.

I'm also using one of the two M.2 slots on the motherboard for a 2TB Samsung 970 Evo that I have passed through to a Virtual Machine.

On this board right now I have 11 PCIe SSD's connected directly to CPU lanes and 11 Hard Disk Drives connected to the 9500-8i Host Bus Adapter through a SAS expander present on my chassis. Power consumption on the entire system with all these components is 280 Watts idle and 609 Watts under a 100% CPU load with some RAM utilised when under load too.

Overall I think this motherboard is fantastic. Often when you start loading a board up with so much stuff you run into edge cases where things are a bit janky but this has been smooth sailing ever since I got it in 2022 and I've continued to upgrade the hardware over time, add more PCIe cards, more storage and so on.

For the operating system I'm running Linux, specifically unRAID 7.0 as of this post. I run a lot of Virtual Machines on this system with devices passed through to those VM's including two ports on the X710-DA4 and both ports on the X550-T2 aswell as a Samsung SSD. The IOMMU groups are great and make breaking things out to VM's very simple and reliable.

I hope this is helpful to anyone thinking about basing a system around this board I certainly recommend it

 

[bugacha]

I hope this is helpful to anyone thinking about basing a system around this board I certainly recommend it

Nice system you have there!

What are you running there if not secret? How do you use all those resources?

 

[Pri]

Nice system you have there!

What are you running there if not secret? How do you use all those resources?

I'm a software developer and I run a SaaS (Software as a Service) business that deals with both B2B and B2C sales. So a lot of the resources here in this server (which is at my home) are for running software that I've written that help with my business operations.

For example the 1TB of RAM and the CPU's X3D cache help massively with a large database that I run on the system where I'm updating and accessing a lot of varied information, using a lot of this memory as cache massively improves transactions per second, to put it into perspective with the X3D cache disabled I can do about 1 million transactions per second and with it enabled that jumps to 1.87 Million, almost a doubling.

In addition I run a lot of VM's on this system which mirrors (to a smaller extent) my live infrastructure where I rent servers all over the world to deliver service to my customers, this allows me to test deploy code and see how the traffic between nodes in the cluster work and such.

The NVIDIA RTX A4000 GPU I'm using for machine learning on the system, to train my own model that I deploy weekly on my live systems and also for local LLM use with my IDE for coding assistance over my local network to my desktop and laptop. I didn't need a huge amount of GPU compute at the time I bought it and with this card being only single-slot it was the main reason I chose it over another more powerful card.

As for the SSD's the 7.68TB Kioxias I'm using for databases and training data, I have them in a RAIDZ1. The 4 x 2TB 980 Pro I'm using just as a disk cache for unRAID which handles incoming data that is destined for the Hard Disk Drives, I have these four SSD's in a BTRFS RAID0. The two SN850's I have in a mirror RAID1 and I use them for VM's and Docker containers.

The two network cards I have in the system I'm using for general connectivity for the main system, the VM's, the dockers and I'm also passing through a few for a pfSense VM which I use as my router/firewall. I have two internet lines 1.2Gb and 5.5Gb (up/down) so pfSense is load balancing etc

 

[OKarlsson]

Hi everyone,

I need your help guys. I have just purchased a Supermicro H12SSL-CT and I am not being able to make it boot or post. Before this mobo I sent back an Asrock RomeD8-2T because of the same issue, no boot, no post.

These are the specs of the rest of the components. AMD EPYC 7532 (thinking this may be the issue). Samsung ram, DDR4 2933 ecc reg. Tested on a H11dsi-nt rev2, working perfectly. Also tried ddr4 2133 ecc reg. Tried motherboards VGA and two different nvidia gtx980. Tried disabling VGA through jumper. Tried two different power supllies: a brand new Corsair HX1200i and an older AX760i

I installed the processor with 1.5 Nm screwdriver. Used a noctua cooler. Tried only one ram stick, various graphic cards, two different power supplies. Fans spin but I get no image. I am able to log in through IMPI but the hardware is not correctly recognized (the processor and ram). I get no sensor readings, no fan readings. Remote console shows nothing. BIOS and firmware are the latest. Post snooping show 03-a1-de-01-35-ad. Depends on the moment. Cleared CMOS to no change. When I turn on the mobo, power ok led blinks and BMC blinks rapidly. Then power ok led turns off. It turns on when I turn the computer on.

Since the Asrock mobo didnt work either, and the rest of the components have already been tested, I am thinking it could be the processor (its second hand, like the ram). Either that or I am missing something with this server motherboards (I am new to this type of servers). DO I need to do something else in the motherboard I may have not done?

What do you think? Any help would be really appreciatted. Thank you

 

[Pri]

Hi everyone,

I need your help guys. I have just purchased a Supermicro H12SSL-CT and I am not being able to make it boot or post. Before this mobo I sent back an Asrock RomeD8-2T because of the same issue, no boot, no post.

These are the specs of the rest of the components. AMD EPYC 7532 (thinking this may be the issue). Samsung ram, DDR4 2933 ecc reg. Tested on a H11dsi-nt rev2, working perfectly. Also tried ddr4 2133 ecc reg. Tried motherboards VGA and two different nvidia gtx980. Tried disabling VGA through jumper. Tried two different power supllies: a brand new Corsair HX1200i and an older AX760i

I installed the processor with 1.5 Nm screwdriver. Used a noctua cooler. Tried only one ram stick, various graphic cards, two different power supplies. Fans spin but I get no image. I am able to log in through IMPI but the hardware is not correctly recognized (the processor and ram). I get no sensor readings, no fan readings. Remote console shows nothing. BIOS and firmware are the latest. Post snooping show 03-a1-de-01-35-ad. Depends on the moment. Cleared CMOS to no change. When I turn on the mobo, power ok led blinks and BMC blinks rapidly. Then power ok led turns off. It turns on when I turn the computer on.

Since the Asrock mobo didnt work either, and the rest of the components have already been tested, I am thinking it could be the processor (its second hand, like the ram). Either that or I am missing something with this server motherboards (I am new to this type of servers). DO I need to do something else in the motherboard I may have not done?

What do you think? Any help would be really appreciatted. Thank you

Sounds like a faulty CPU. It does happen, I had one recently which was faulty (EPYC). I assume you are following the 1-2-3 to install pattern when screwing the CPU socket down as that does actually matter with EPYC to get it to sit flat in the socket.

 

[OKarlsson]

Sounds like a faulty CPU. It does happen, I had one recently which was faulty (EPYC). I assume you are following the 1-2-3 to install pattern when screwing the CPU socket down as that does actually matter with EPYC to get it to sit flat in the socket.

Hi Pri, thanks for your comment. Yes, I am following the correct order for locking the socket. I was very carefull while installing it. Is there anything I should be aware of regarding the motherboard, any setting I may have missed? But yeah, I agree, to me looks like the CPU but I want to make sure I have setup the motherboard propertly. THank you

 

[Pri]

Hi Pri, thanks for your comment. Yes, I am following the correct order for locking the socket. I was very carefull while installing it. Is there anything I should be aware of regarding the motherboard, any setting I may have missed? But yeah, I agree, to me looks like the CPU but I want to make sure I have setup the motherboard propertly. THank you

There is nothing on the motherboard that would stop it posting/booting to my knowledge.

One last thing to check, make sure the CPU is inside the carrier card the right way, sometimes they fall out and are placed back in the wrong way round. You'll find a gold triangle on the CPU which you can match to the CPU socket (which also has a black printed triangle) to make absolutely certain the CPU is in the carrier in the correct orientation.

If that's good too then I would replace the CPU as you've ruled out all other hardware.

 

[OKarlsson]

If I recall correctly I checked that througly but I will make sure, thanks for your input

 

[LolSalad]

Hi everyone,

I need your help guys. I have just purchased a Supermicro H12SSL-CT and I am not being able to make it boot or post. Before this mobo I sent back an Asrock RomeD8-2T because of the same issue, no boot, no post.

These are the specs of the rest of the components. AMD EPYC 7532 (thinking this may be the issue). Samsung ram, DDR4 2933 ecc reg. Tested on a H11dsi-nt rev2, working perfectly. Also tried ddr4 2133 ecc reg. Tried motherboards VGA and two different nvidia gtx980. Tried disabling VGA through jumper. Tried two different power supllies: a brand new Corsair HX1200i and an older AX760i

I installed the processor with 1.5 Nm screwdriver. Used a noctua cooler. Tried only one ram stick, various graphic cards, two different power supplies. Fans spin but I get no image. I am able to log in through IMPI but the hardware is not correctly recognized (the processor and ram). I get no sensor readings, no fan readings. Remote console shows nothing. BIOS and firmware are the latest. Post snooping show 03-a1-de-01-35-ad. Depends on the moment. Cleared CMOS to no change. When I turn on the mobo, power ok led blinks and BMC blinks rapidly. Then power ok led turns off. It turns on when I turn the computer on.

Since the Asrock mobo didnt work either, and the rest of the components have already been tested, I am thinking it could be the processor (its second hand, like the ram). Either that or I am missing something with this server motherboards (I am new to this type of servers). DO I need to do something else in the motherboard I may have not done?

What do you think? Any help would be really appreciatted. Thank you

Which motherboard revision do you have?

 

[RolloZ170]

I am able to log in through IMPI but the hardware is not correctly recognized (the processor and ram)

updated by POST, no POST = old data.

Post snooping show 03-a1-de-01-35-ad.

if DE AD comes in series it means indeed "DEAD". this comes afaik from AGESA and is a final stop.
CPU