Does anyone have Supermicro H12SSL series motherboard experience?

[mattlach]

Hey everyone,

It's time to upgrade the old home server, and I think I have settled on a 24+ core 3rd gen Epyc (Milan) on one of Supermicro's H12SSL motherboards (I'll probably go with the H12SSL-NT)

Just a quick question about these for those with experience:

1.) If I were to buy one new off of Amazon today, am I likely to get a recent enough firmware/BIOS that a Milan CPU will work?

2.) If I do need to upgrade the firmware/BIOS before using a Milan CPU, can this be done via the BMC without a compatible CPU installed? ...or should I also be shopping for a bargain basement Naples CPU to use for flashing purposes?

3.) Any tips on how to avoid getting a vendor locked CPU when shopping used?

I have a narrow (~1 week) window of downtime in order to do the upgrade and get everything up and running again, and I'd very much like to avoid the usual "start project, discover problem, order more parts to fix problem, wait for parts to arrive" cycle.

I appreciate any suggestions!

 

[jei]

Check https://forums.servethehome.com/ind...-nt-h12ssl-ct-boards-notes-experiences.33468/

1) Amazon can be anything I guess. I bought H12SSL-NT 2 weeks ago from a German retailer and it came with 2.6a BIOS (27-Sep-2023).

2) Yes you can update the BIOS without CPU.

3) I can recommend tugm4470 | eBay Stores

 

[sam55todd]

...2) Yes you can update the BIOS without CPU...

I had an experience with trying to update BIOS on SuperMicro MBs X13 series (without CPU because I've received board first but had no CPU at that time) - New version of BIOS is simply sitting in validation/staging state (waiting for successful boot) before being committed as working version.
Therefore I wouldn't call it completed update.
After receiving first CPU (ES version which as it turned out was not compatible) and installing it - this staging area bios still wasn't promoted into live copy (obvious reason: no successful boot therefore no validation it's operational).
Only after getting (and installing) right CPU - new bios was promoted into live state after first fine boot.
But who knows - perhaps H12SSL (which is previous generation vs X13) has different firmware update workflow logic with not so strict rules.

 

[mattlach]

Check https://forums.servethehome.com/ind...-nt-h12ssl-ct-boards-notes-experiences.33468/

1) Amazon can be anything I guess. I bought H12SSL-NT 2 weeks ago from a German retailer and it came with 2.6a BIOS (27-Sep-2023).

2) Yes you can update the BIOS without CPU.

3) I can recommend tugm4470 | eBay Stores

Thank you sir. I appreciate the info.

3) I can recommend tugm4470 | eBay Stores

Thanks for this recommendation. I had previously heard that this seller was legit, though buying absolutely anything from a Chinese seller makes me nervous as all hell. I guess with a CPU there is little they can do, but anything with firmware is risky. I mean, you can always flash it the instant you get it, but who knows what else may be hiding in there...

Were shipping times reasonable when you ordered from tugm? Or is this a "several weeks on the slow boat from China" experience? I would probably need to have the part in my hands within 2 weeks.

 

[mattlach]

oh, and one more question. I have three orange Threadripper torque wrenches that came with 3000 series Threadrippers. Are they going to have the same torque specs as the Epyc ones?

 

[sam55todd]

though buying absolutely anything from a Chinese seller makes me nervous as all hell

Thankfully going not directly but via eBay is generally saving from a lot of pain (with their consumer protection policies)
I was getting easy refunds for DOA or not "as described" equipment without much of a hassle.

P.S. Ohh just realized you might be talking about security risks due to possibility of flashing extra malicious code into firmware.
In this case yes - this is potential nightmare (and BIOS is not the only firmware on MBs, without even going into possible other chips replacement with extra logic).

 

[mattlach]

P.S. Ohh just realized you're talking about security risks due to possibility of flashing extra malicious code into firmware.
In this case yes - this is potential nightmare (and BIOS is not the only firmware on MBs, without even going into possible other chips replacement with extra logic).

Indeed. I always reflash what I can as a non-electrical engineer without access to specialty chip programming tools whenever I buy second hand electronics, but to your point, there is some stuff us regular users can't quite access, at least not easily.

Some Supermicro boards have LSI/Broadcom SAS adapters in them which have their own firmware, and those I know how to flash as well, but who the hell knows what else might be on them, and what obscure flashing tool might be needed, and which firmware to flash even if you had it. Or if there are chips that need to be pulled and flashed in a chip programming tool.

It's a brave new world out there, and I trust no one, but there is only so much you can do as a reasonably normal person

 

[hmw]

oh, and one more question. I have three orange Threadripper torque wrenches that came with 3000 series Threadrippers. Are they going to have the same torque specs as the Epyc ones?

yes

 

[jei]

I had an experience with trying to update BIOS on SuperMicro MBs X13 series (without CPU because I've received board first but had no CPU at that time) - New version of BIOS is simply sitting in validation/staging state (waiting for successful boot) before being committed as working version.
Therefore I wouldn't call it completed update.
After receiving first CPU (ES version which as it turned out was not compatible) and installing it - this staging area bios still wasn't promoted into live copy (obvious reason: no successful boot therefore no validation it's operational).
Only after getting (and installing) right CPU - new bios was promoted into live state after first fine boot.
But who knows - perhaps H12SSL (which is previous generation vs X13) has different firmware update workflow logic with not so strict rules.

This could very well be the case as BIOS 2.6a already supports Milan.

Were shipping times reasonable when you ordered from tugm? Or is this a "several weeks on the slow boat from China" experience? I would probably need to have the part in my hands within 2 weeks.

"Order date:Jan 13, 2023 Delivered on Wed, Jan 18." This is China -> Finland. At the time there was an offer for STH members to get priority shipping. Maybe still is.

 

[RolloZ170]

3.) Any tips on how to avoid getting a vendor locked CPU when shopping used?

the seller should mention this.
attention if the price is too good.
if the seller states "every CPU is tested on our Dell server before shipping" avoid buying.

add.info on most H12/X12/H13/X13 motherboards:
these use Root of Trust (RoT) Firmware system.
if you turn on the power(Standby, not pwr btn) the CPU is helt in reset even if you pwr on.
the BMC self-checks its FW. then the BMC loads the content of BIOS chip and does the validation,
it is signed with supermicro private key/cert, the check is done with a matching public key/cert,
if you want to modify the FW you have to re-sign the BIOS with private key/cert, i doubt you will get it from SM.

the BMCs FW is protected by a write protected bootloader block.
only the BMC can flash the BIOS, the DOS/EFI untility just uploads the BIOS to the BMCs(ASPEED) flashdisk,
initiates a reboot and lets the BMC do the job.
No working BMC, no BIOS update (even does not work if you forgot passwords)
after successfully POST of lately flashed BIOS you can see the new BIOS version in the BMCWebGUI screen.

 

[mattlach]

Thanks for all the help guys.

I have an Epyc 7543 (32C/64T) incoming from tugm, an H12SSL-NT incoming from Amazon and 512GB of registered DDR4-3200 coming as well.

Wish me luck!

 

[mattlach]

The CPU from Tugm just arrived today. Pretty impressive shipping time!

I'm going to attempt some benchtop testing to make sure everything is working and it is performing right before upgrading the old server.

With desktop parts I've usually done this by reading reviews and running matching benchmarks to replicate their results, and then done a day or two of mprime/prime95 for stability, but I am having a difficult time finding good tests to run on this platform.

Any recommendations for what to do for an out of box baseline performance test to make sure everything is working right?

 

[oneplane]

What did you end up paying for all of that, and what kind of chassis, cooler and power supply are you using? Sounds like a neat setup!

 

[mattlach]

What did you end up paying for all of that, and what kind of chassis, cooler and power supply are you using? Sounds like a neat setup!

hehe.

I've been trying to not add it all up to avoid giving myself a heart attack, but sure, I'll give it a try

- Epyc 7543 (32C/64T, 2.8Ghz Base, 3.7Ghz Boost) - $1,369.00 + sales tax (used from tugm4470 on ebay)
- Supermicro H12SSL-NT-O - $636.99 + sales tax (new on Amazon)
- 8x Hynix 64GB Registered DDR4-3200 - $879.92 + sales tax (used on ebay from atechcomponents)
- Supermicro SNK-P0064AP4 92mm Socket SP3 CPU cooler for 4U Cases - $78.49 + sales tax (new on amazon)

So, I guess that adds up to $2967.40 (yikes)

The dual socket E5-2650 v2 is getting really old though, so it was kind of unavoidable.

I could probably have done better by buying a used HP or Dell server, but I am never doing that again. I learned that lesson a few years back with my HP DL180G6 Dreamliner. Thing sounded like a jumbo jet taxiiing out on the runway, and would not be tamed enough for home use.

Anyway, the plan is to do a drop in replacement in my slightly customized Supermicro SC-846 4U chassis. All I did to that was to replace the fan wall and its noisy 80mm fans with three 120mm high speed fans to tame the beast.

My existing power supplies should be sufficient according to my math. The old E5-2650 v2's clocked in at 95W each, whereas the EPYC has a TDP of 225w, so its 30W more at full load, but I'm pretty sure I have plenty of lee-way in my dual Supermicro PWS-920P-SQ's, especially since it is WAY more efficient at idle, where it will spend most of its life.

Of course, there could be some nasty compatibility surprise, but I feel like I can make it work. If my old Supermicro X9DRI-F can run off of this thing, the H12SSL should do so as well. (fingers crossed)

X9DRI-F takes standard ATX power + two 8pin EPS power plugs. The H12SSL-NT is the same.

Worse comes to worse, and I go slightly above 920w at load, I'll just have to drop the full redundant capability, and just run cumulative power, which is fine for my application, where I rarely ever see anything close to peak load.

If I lose a PSU it starts beeping to high heavens, and then I'd just shut it down and order a replacement.

Actually, come to think of it, I still have the dual 1200W power supplies the server shipped with. I didn't want to use them because they were a but on the loud side, but now the rack is 75 ft away from my office, so I don't really care anymore. I could just pop 'em back in if it comes to that. I'm going to have to measure power use when power it up to make sure.

 

[mattlach]

Decided to do some basic assembly:

This is the part that no matter how many times I do it on a Threadripper (and now EPYC) always scares the shit out of me:



4094 hair-fine pins that unlike some others are irreparable if touched.

With my lock I'd drop the CPU on them, which is probably why they include that nifty secondary plastic cover.



And a few minutes later. I wanted to do some benchtop testing today, but apparently I no longer have a good PSU in my spare parts bin. I'm going to have to yank the one out of my testbench machine, but I just don't feel like doing it tonight.

I'm a little bit concerned about the clearance behind PCIe slot Slot three with those weird Supermicro plastic M.2 clips sticking up, but I'm hoping it will be fine. I'm also considering putting a couple of slim heatsinks on those m.2 drives to ensure they don't get too toasty, but the Amazon store page says they only stick up by ~3mm, so hopefully I'll be OK. Those are going to be my mirrored boot drives (Using ZFS). I'm going to be using at least 5 maybe 6 of those slots, so they can't be blocked. This depends on if I can get the funky SlimSAS ports to run my Optane u.2 drives. If I can, then I don't need my u.2 riser card. 3 of the slots will have 16x cards, the rest 8x cards.

It's a little weird to see such a petite CPU cooler on a 225W TDP CPU, but we are not overclocking here. As long as we stay under TJMax at full load I'll be happy. Even better if I can get the full advertised boost out of it.

 

[sth]

Nice build. I’m using that cooler on a similar 75F3 in a SC846 and it’s plenty capable. I think there’s value running a memtest to verify all those pins are connected, it took a few days for 512GB to complete a few passes. The premium version tests ECC RAM too.

 

[i386]

View attachment 33188

In case you encounter random crashes try other m.2 ssds. I had nothing but bad luck with samsung 990 pro ssds as boot devices (https://forums.servethehome.com/ind...amsung-980-and-990-with-windows-server.39366/)

 

[sth]

With Supermicro I stick with the recommended hardware to avoid those kinds of issues. I’ve had odd random issues with RAM, SSDs and plug in PCI expanders over the years that have been hard to debug and not been supported by SM.

 

[mattlach]

In case you encounter random crashes try other m.2 ssds. I had nothing but bad luck with samsung 990 pro ssds as boot devices (https://forums.servethehome.com/ind...amsung-980-and-990-with-windows-server.39366/)

Interesting, thanks for the heads up. I actually bought these 980 Pro's specifically for the task, as I presumed they would be more reliable than the "Inland Premium" drives I bought in bulk a few years ago and have been using in my current server for various tasks (but not booting from, as it doesn't support that)

Mind sharing what you did wind up using that worked for you?

I vaguely remember the 990 Pro having some sort of firmware issue when it was first launched that required a new firmware to fix it. I can't remember the details though. I wonder if that was the issue.

 

[mattlach]

With Supermicro I stick with the recommended hardware to avoid those kinds of issues. I’ve had odd random issues with RAM, SSDs and plug in PCI expanders over the years that have been hard to debug and not been supported by SM.

My experience has been good compatibility wise and random hardware with my X9DRI and my X8DTE before it, but I was cognizant of potential memory issues and tried the Supermicro QVL, only to find that there was very little on there, and nothing in the size I needed.

In my current X9DRI I have several random PCIe devices that just work. One dual PCIe 8x to two u.2 adapter, one old Intel X520 NIC, and three Asus x16 to four m.2 risers. All of them work well (though getting the bifurcation settings right was tricky, especially due to a bios bug resulting in the PCIe slots being mislabeled causing some frustration)

Fingers crossed that they work out alright. I'm hopeful that if it worked in my X9DRI, it will work in this board. Only new devices I plan on using are the boot m.2 drives. I explicitly got the 980 Pro's as I figured they would be more reliable than what I usually get, and I didn't feel like buying used enterprise SSD's (used drives just never feel right no me)