Supermicro H13SSL-N and fans oscillating between low and high speed

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
X

xBelladonna

New Member
Aug 15, 2023
4
2
1
Hi everyone,

I've just build a hell of a monster with an Epyc 9334 QS in a Supermicro H13SSL-N. Not really knowing how all this worked yet, I grabbed some Noctua fans, as I had on my last build, but it seems even adjusting the thresholds either does nothing, so they keep oscillating, or has the fans within range (no longer critical) yet the speed is still on full. I have tried toggling it between different speeds in the web interface itself and adjusting with IPMI. Nothing seems to make a meaningful difference.

Would anyone have any idea of whether the H12/13 series of motherboards can be made to behave with Noctua fans or am I just SOL and need to get different fans? If the answer is different fans, would anyone have any suggestions on what would be fast enough to remain above the boards lower critical threshold?
Supermicro's own offerings, maybe? Supermicro 120mm Axial Fan (FAN-0124L4)

Willing to provide as much additional info as I can.
 
Last edited:
  • Like
Reactions: michel333
C

CreoleLakerFan

Active Member
Oct 29, 2013
485
181
43
M

mirrormax

Active Member
Apr 10, 2020
227
86
28
the solutions above should work fine had to do that myself in the past. iam surprised they havnt fixed this yet and added fan curves like other vendors have even on their server boards.
curious how the QS cpu is running, max clocks single core and all core?

iam considering the same cpu for my next workstation, or a xeon qs/es maybe..
 
X

xBelladonna

New Member
Aug 15, 2023
4
2
1
mirrormax said:
the solutions above should work fine had to do that myself in the past. iam surprised they havnt fixed this yet and added fan curves like other vendors have even on their server boards.
curious how the QS cpu is running, max clocks single core and all core?

iam considering the same cpu for my next workstation, or a xeon qs/es maybe..
Click to expand...
Hi there, I've been working on this server for a week and a bit since this thread and I've found that the Noctua NF-F12 Indusstrial PPC 3000 RPM version work really well to keep the case nice and cool, while also spinning at a rate fast enough for this cursed BMC. They are... not quiet at any real speeds but at lower speeds they're good enough that I can sleep in my studio apartment next to this thing.

I've noticed that the fan speed needs to be set to "Optimal Speed" to prevent it from alternating back and forth between a rough speed increase and back to baseline once it gets up to a certain temperature, something like 45-50 degrees. As far as the performance goes, it kicks ass and I haven't had any issues with it yet. It's hard to identify just based on the data the chip identifies itself with, but that's to be expected. Just save the CPUs details somewhere safe. Also keep in mind I've only been running this gently for 2 weeks, not any extensive period of stress testing.

Just for kicks, this is what it did in Cinebench (over the iKVM it can't even display the full window lol)
Untitled.png

It's done in about 15 seconds, it's insane. And, this is with WINE. I'll see if I can do some more benchmarks and get clock speeds back to you.

I have been looking into porting openbmc to this board, since it looks like Supermicro actually bastardized and licencified the openbmc framework for the new BMCs on the H12/13 series. It's not trivial to reverse engineer but it should be possible. Let's see if someone better than me gets to it first.
 
  • Like
Reactions: gianasisters
N

nexox

Well-Known Member
May 3, 2023
1,311
609
113
One thing I have done on systems that wouldn't let me set the fan thresholds is use the Noctua splitter wire which sends PWM to both fans, but only RPM sense from one fan, then put a smaller fan on the RPM sense side, like a 40mm on the chipset or VRMs.
 
S

sand

New Member
Jan 18, 2023
7
0
1
That thing is fast!

Have you run into any problems with the QS version in your testing?

I'm tempted to buy one on Ebay sellers seem a bit shady or new.
 
X

xBelladonna

New Member
Aug 15, 2023
4
2
1
Hi, apologies for the late reply. I have been running into some really tedious VFIO issues with USB PCIe expansion cards, and have had to return a few. It seems the Renesas uPD720201 doesn't work so well with this motherboard and has issues with power delivery even with an extra power connector, so do keep that in mind. VIA controllers are also very bad news but that appears to be a common, well documented issue anyway. However, the uPD720202 (which is the same controller used on the board itself) works just great, and the Startech 4-controller card works perfectly.

I have also ended up using a Noctua splitter to break out the separate fan zone on headers A and B this board seems to have to drive all 3 of my case fans instead of just intake over the backplane, which moves more air through the other components in the case like NVMe drives.

So far I haven't had any issues with the QS CPU, but I've been working out an issue with my RAM over the last week and I think I've narrowed it down to 2 bad DIMMs. I'm in the process of RMAing the RAM and getting replacements sent out. Hopefully everything ends up working, but it's still possible it could be something to do with the memory controllers on the SOC, so I can't really say yet. One thing is for sure, make sure you are using a calibrated torque wrench to do up the 7 heatsink screws and use several passes in a star pattern. Turns out 6000 pins over the area of 1.5 threadrippers is kinda sensitive to mounting pressure and plane warp. Once I've finally managed to bring this server to its Final Form(tm) I will be able to do more complete testing.
 
S

sand

New Member
Jan 18, 2023
7
0
1
excellent, Thanks for the reply. I'll roll the dice with the QS from china on ebay. Going to get the GIGABYTE MZ33-AR0
 
M

mikegrok

New Member
Feb 26, 2023
16
4
3
Alabama, USA
I have the
sand said:
excellent, Thanks for the reply. I'll roll the dice with the QS from china on ebay. Going to get the GIGABYTE MZ33-AR0
Click to expand...
By the way, you will get collisions between your ram sticks and your PCIe cards with that motherboard.

I have the H13SSL-NT and the genoa epyc 9124 with 96gb in 3 sticks.

By the way, the first siena boards and chips started shipping this week, but not in atx format.
 
S

sand

New Member
Jan 18, 2023
7
0
1
mikegrok said:
I have the


By the way, you will get collisions between your ram sticks and your PCIe cards with that motherboard.

I have the H13SSL-NT and the genoa epyc 9124 with 96gb in 3 sticks.

By the way, the first siena boards and chips started shipping this week, but not in atx format.
Click to expand...
ack. What kind of collisions will I get? I was planning on only using 12 of the ram slots... Was hoping as long as I stayed away from the other 12 slots I'd not have any issues using four pcie cards. I can't find anyone documenting that on the mz33 (there are some issues with mz32 i see).

For your setup aren't you losing a lot of ram bandwidth by only using 3 sticks? one reason I wanted an genoa epyc is the 12 channels of ddr5.
 
M

mikegrok

New Member
Feb 26, 2023
16
4
3
Alabama, USA
Most RDIMMs are incompatible with dual dimms per slot, and are optimized for single dimm per slot, and already doing the interleaving that you can achieve by installing matched dimms, which achieves that extra 3% speed. I just bought the commodity rdimms on the supermicro approved ram page. When I was looking 2 dimms per channel ram was 5x the price per gig of 1 dimm per channel at 64GB, and only available on 64GB and larger sticks. When I was last looking, 64GB dimms were 2.5 times the price per gig of 32GB dimms.

I have 3 channels, to get equivalent transfer rates with 2 channels I would need ddr5-7200. My main issue with the ram is that about 3 years after I buy the system, I start running out of ram, and the computer doesn't hold any more. The difference between enough ram and not enough ram is much more significant than the speed difference from having a bunch of channels. In theory 12 channel ram is 6 times faster than 2 channels. However not having to use virtual memory is 100 times faster than most fast M.2 SSDs. Also I am running 1/8 the max number of CPU cores, I don't think that they are bandwidth starved.

The other thing about ram, is that it is used or it isn't, and there is very little advantage to having more ram than you are utilizing. Last I checked with all of the software I was running before I started adding VMs, I was using 20GB of ram. I am going to add a 16GB truenas core VM, and that still leaves plenty of ram left for experiments before I start running out. And if I start running out, I can just buy more, I have the empty slots.

Ram price will drop, and capacity will increase with time. By the time I fill up all of my slots, I can probably buy higher capacity ram at a much cheaper price.

If you are balking at the price, look at the epyc 8004 series in my earlier post, they are more for connectivity than bulk computation, and are effectively 2/3 an epyc 9004 at 1/3 the price, with about 3x the connectivity of a Ryzen, but at a similar price point.
 
  • Like
Reactions: drdepasquale
M

mikegrok

New Member
Feb 26, 2023
16
4
3
Alabama, USA
xBelladonna said:
Hi, apologies for the late reply. I have been running into some really tedious VFIO issues with USB PCIe expansion cards, and have had to return a few. It seems the Renesas uPD720201 doesn't work so well with this motherboard and has issues with power delivery even with an extra power connector, so do keep that in mind. VIA controllers are also very bad news but that appears to be a common, well documented issue anyway. However, the uPD720202 (which is the same controller used on the board itself) works just great, and the Startech 4-controller card works perfectly.
Click to expand...
PCIe 5 is very finicky. Put cards in the slots, for anything off board, use a cable to a mcio port. If you must make the pcie slot remote, it will probably need a retimer on the card.

This is why they sell this board:

It has 10 x8 mcio ports and 2 pcie 5 x16 slots.

They then put it in this server:

AS -2015HS-TNR | 2U | A+ Server | Products | Supermicro

2U UP Hyper with 12 hot-swap 3.5" drive bays and up to 8 PCIe 5.0 slots
www.supermicro.com
which allows you to connect 12 u.2 drives with up to pcie5 x4 connectivity to the motherboard.
You can of course put it in your own server to roll your own solution.
 
Last edited:
M

mikegrok

New Member
Feb 26, 2023
16
4
3
Alabama, USA
mikegrok said:
If you must make the pcie slot remote, it will probably need a retimer on the card.
Click to expand...
If you look at this board:
www.gigabyte.com

ME03-PE0 (rev. 1.x) | Server Motherboard - GIGABYTE Global

Single AMD EPYC™ 8004 Series Processors6-Channel RDIMM DDR5, 6 x DIMMs2 x 1Gb/s BASE-T LAN ports via Broadcom® BCM57202 x MCIO 8i connectors with PCIe Gen5 ...
www.gigabyte.com www.gigabyte.com

Those chips on the motherboard by the far pcie4 slots are retimers. You may be able to use passive cards in this board for remote pcie.
 
R

RolloZ170

Well-Known Member
Apr 24, 2016
6,681
2,067
113
mikegrok said:
If you look at this board:
www.gigabyte.com

ME03-PE0 (rev. 1.x) | Server Motherboard - GIGABYTE Global

Single AMD EPYC™ 8004 Series Processors6-Channel RDIMM DDR5, 6 x DIMMs2 x 1Gb/s BASE-T LAN ports via Broadcom® BCM57202 x MCIO 8i connectors with PCIe Gen5 ...
www.gigabyte.com www.gigabyte.com

Those chips on the motherboard by the far pcie4 slots are retimers. You may be able to use passive cards in this board for remote pcie.
Click to expand...
no, that are switches. slot 2&4 are not available if you want Slot1&3 x16
 
  • Like
Reactions: vvkvvk
G

gianasisters

New Member
Dec 4, 2023
1
1
1
xBelladonna said:
Hi there, I've been working on this server for a week and a bit since this thread and I've found that the Noctua NF-F12 Indusstrial PPC 3000 RPM version work really well to keep the case nice and cool, while also spinning at a rate fast enough for this cursed BMC. They are... not quiet at any real speeds but at lower speeds they're good enough that I can sleep in my studio apartment next to this thing.

I've noticed that the fan speed needs to be set to "Optimal Speed" to prevent it from alternating back and forth between a rough speed increase and back to baseline once it gets up to a certain temperature, something like 45-50 degrees. As far as the performance goes, it kicks ass and I haven't had any issues with it yet. It's hard to identify just based on the data the chip identifies itself with, but that's to be expected. Just save the CPUs details somewhere safe. Also keep in mind I've only been running this gently for 2 weeks, not any extensive period of stress testing.

Just for kicks, this is what it did in Cinebench (over the iKVM it can't even display the full window lol)
View attachment 31372

It's done in about 15 seconds, it's insane. And, this is with WINE. I'll see if I can do some more benchmarks and get clock speeds back to you.

I have been looking into porting openbmc to this board, since it looks like Supermicro actually bastardized and licencified the openbmc framework for the new BMCs on the H12/13 series. It's not trivial to reverse engineer but it should be possible. Let's see if someone better than me gets to it first.
Click to expand...
Is any chance for Single Core benchmark? I need consider to switch from my 3970x. Kindly request...Thx in advance.
 
  • Like
Reactions: ServerSemi
mattlach

mattlach

Active Member
Aug 1, 2014
404
163
43
This has been an issue for me every time I've set up a new Supermicro server board since the ~x9 era.

It's usually mostly an issue from cold, and once the system warms up, the oscillation stops. I can usually make it go away all together by playing with the fan modes in BMC, but I can't remember the exact ones that work for me.
 
You must log in or register to reply here.
Top Bottom