supermicro e300-9d-8cn8tp Cooling options (homelab)

[Rolando]

At the beginnings the 3 things I was looking for build my home lab was:

1. Small form factor.
2. As silent as possible (I don’t have a server room).
3. High RAM capacity (I work a lot with virtualization)

So, I got a e300-9d-8cn8tp from supermicro for my home lab because the support for lots of RAM (currently using 256GB ) and the first thing I did was to replace the noisy stock fans for a set of Noctua NF-A 4x20 with great results (from the noise factor), but because the CPU uses a passive heatsink I had a lot of situations where the CPU temperature reached more than 90 degrees Celsius for the CPU and the PCH with an error message appeared in the IPMI regarding this (I already upgraded the IPMI just to double check if there was a issue with the interface reading the temperature sensors).

So the passive cpu cooling when using the Noctua fans is not enough for keep the system working properly.
And the main challenge is that the socket that this Xeon-D uses is not very common like for get an aftermarket cpu fan and change from passive to active cooling for the CPU. After some research I found this part number: SNK-C0111AP4L which is the heatsink with a fan provided by supermicro, the main concern I had is that the height of this will not fit the 1U form factor of the e300-9d-8cn8tp system.

So after receive the cpu fan and install it, the temperature of the CPU is not higher than 75 degrees Celsius under high load, now the issue is that I can’t use the cover of the case which also affects the airflow of the other fans and will expose the system to other stuff like dust as this is sitting by my desk. So is time to search for other case option.

After research a little bit I found that the gaming community have a lot of interesting options for cover the first two requirements I have which are small form factor and silence, which are the mini-itx systems. There are several cases and cpu cooling options for that, but then I found that the motherboard I have is not mini-itx is a flex-ATX which for my bad luck will not fit any of the small form factor cases available in the market .

So at the end I think I have two options:

1.- Keep looking for a small form factor case that fits the flex-ATX motherboard (most of the options I found are to big considering the size of the current case I have). Also I found some information of how to fit a Noctua nh-l12s in this socket, so I think that I could use that.

2.- Sell the barebone and keep the RAM and storage and get a X11SDV-8C-TLN2F along with a good looking mini-itx case like the ghost s1 with the Noctua Nhl12s Cpu fan.

I like more the option two, however it could take a while as here in Peru is not very common to have this systems. I manage to get the parts I needed during a business trip to US And plan to have a another business trip in December where I could get the parts I need based on which option I take.

Any experience, recommendation or comment with this situation is more than welcome.

Thanks!

 

[Francis Augusto Medeiros]

Hello!

I don't know how and if you solved your problem, but I installed a similar motherboard directly on a Fractal Design case: Building a Supermicro-based PC for ESXi and FreeNAS

I don't know if that's small enough for you, though.

I am thinking of going the opposite way, that is, moving to an E300, so it helped me a lot to read your experience. I was hoping that the active cooling would fit the E300, though now you confirm it doesn't, which is a bummer for me.

I was wondering, if you remove the fan from your SNK-C0111AP4L, do you get higher temperatures when having your setup on the E300 chassis compared to the original heatsink?

Best,

Francis

 

[Rolando]

Hello Francis, I did tried without the fan and the system overheated and powered off after a couple of minutes of being started. E300 runs just fine only if you are ok with the loud factory fans, otherwise I recommend to keep the mini-itx form factor, I wrote a blog about this adventure: HOME LAB PART 5: The Ultimate Small Form Factor Server for Home Lab

 

[ullbeking]

Hello Francis, I did tried without the fan and the system overheated and powered off after a couple of minutes of being started. E300 runs just fine only if you are ok with the loud factory fans, otherwise I recommend to keep the mini-itx form factor, I wrote a blog about this adventure: HOME LAB PART 5: The Ultimate Small Form Factor Server for Home Lab

How is the noise in this setup in your configuration with this rack cabinet?

I'm massively downsizing my hardware and studio workshop, and I would like to move a small amount into my home homelab and run something like this:

• 1U Supermicro A1SRi-2558F router + firewall (quiet)
• 1U 1p D-2100, E-2100, or E-2200 server for computer, hosting, etc (designed for hopeful quietness)
• 1U Supermicro A1SRi-2758F file server or NAS controllers (quiet)
• 2U Lenovo SA120 DAS (12x 3.5" HDD) (maybe loud? need to do research, this is the scary part)

 

[Rolando]

1U form factor is not very friendly when considering noise; however, when I used the Xeon D 1500 series, as the TDP is lower than the Xeon D 2100 the Noctua fans did just fine, so that's an option too. Another option could be to go with AMD EPYC 3000 as the TDP is also lower than the Xeon D 2100 series. I didn't have the chance to try the EPYC 3000 in a 1U case but should be fine.

How is the noise in this setup in your configuration with this rack cabinet?

I'm massively downsizing my hardware and studio workshop, and I would like to move a small amount into my home homelab and run something like this:

• 1U Supermicro A1SRi-2558F router + firewall (quiet)
• 1U 1p D-2100, E-2100, or E-2200 server for computer, hosting, etc (designed for hopeful quietness)
• 1U Supermicro A1SRi-2758F file server or NAS controllers (quiet)
• 2U Lenovo SA120 DAS (12x 3.5" HDD) (maybe loud? need to do research, this is the scary part)

 

[aivxtla]

If considering the EPYC 3000 series here’s my experience at least: I went from a X10SDV-TLNF4 D-1541 board to a M11SDV EPYC-3251 board in my Supermicro 1U short depth chassis. The EPYC 3251 with PUE-2 enabled the Fans run at ~1,500 RPM and I can even get away with a single CPU facing 40 mm fan with idle temps at 38-40C, full load temps hit around 75-80C max I think with 2 40mm fans at ~6,000 RPM, didn’t check max load temps with single fan though. The D-1541 would hit 50 just idling and the same fans at 3,000-4,000 RPM. One of the reasons why SuperMicro’s EPYC 3000 models perform better thermally than their equivalent D-1500 variants, aside from lower power draw, is due a larger heatsink with 2 heat pipes.

There’s also the EPYC3201 which is only 30 Watts and also an 8 core variant.

If you want 10Gbe there are the 3251 and 3101 boards from Asrock.

If you need D-2100 models I’d probably consider a larger case or faster fans (at the cost of greater noise)

Before taking the expensive route and switching boards I’d look at the 0100L4 Nidec fans from Supermicro 8,500 RPM peak and ~35 dBm noise at max. The Noctua 40mm fans aren’t really sufficient at full load from my experience.

 

[Francis Augusto Medeiros]

Hello Francis, I did tried without the fan and the system overheated and powered off after a couple of minutes of being started. E300 runs just fine only if you are ok with the loud factory fans, otherwise I recommend to keep the mini-itx form factor, I wrote a blog about this adventure: HOME LAB PART 5: The Ultimate Small Form Factor Server for Home Lab

Thanks a lot Rolando.
I am considering buying 2U server chassis to use it. Maybe I will keep the Noctua fan on the CPU, or maybe I'll just screw the original fan back on the system. Nice reading your adventure!

 

[ullbeking]

1U form factor is not very friendly when considering noise; however, when I used the Xeon D 1500 series, as the TDP is lower than the Xeon D 2100 the Noctua fans did just fine, so that's an option too. Another option could be to go with AMD EPYC 3000 as the TDP is also lower than the Xeon D 2100 series. I didn't have the chance to try the EPYC 3000 in a 1U case but should be fine.

Thank you @Rolando !

My main question

I'm particularly interested in that 6U rack cabinet. I've already written to the UK distributor. I'm going to do some measurements, and I hope that it will be small enough for a home lab literally in the home. I'm thinking of using it for a primary backup server and a secondary backup server, so noise should be not too much of a problem because the servers will be powered on only when backups operations are in progress. It's all coming together as I think about it more.

Secondary questions

The D-1500 series is wonderful indeed. I have an X10SDV-4C-TLN2F that I hope is quiet enough (I haven't experimented with it enough yet).

I've been a huge fan of the Atom C-2000 servers, e.g., the A1SRi-2758F. They are very quiet, and they are powerful enough for home websites, and are local development servers for developing software with lightweight build infrastructure. Of course this is all on the condition that the Erratum AVR.54 boot failure defect has been fixed by Supermicro or you have a recent enough revision.

I'm also looking at the EPYC 3000, D-2100, D-2200, E-2100, and E-2200 servers, but I have enough in the D-1500 and C-2000 to hold me over for the time being.

@aivxtla has some fascinating things to say about all of this. I'm looking forward to considering their thoughts in more detail.

NAS chassis

The main issue for me and my family is the noise of the NAS chassis. As for as storage is concerned there's an interesting thread about the U-NAS 810 series of chassis specifically designed for desktop NAS'es.

 

[nivedita]

The D-1500 and M11SDV boards can definitely be made quiet. As another data point, I'm currently running an M11SDV-8C-LN4F in the CSE-101F (little brother of CSE-300). I have three fans in it, two FAN-0100L4 (8500RPM) and one FAN-0061L4 (12500 RPM according to specs, but they run at 13500RPM at full speed). The 61L4 idles at 1000RPM (i.e. completely silent), and at full load ramps up to 6-6.5k RPM with CPU temp @ 65C. I actually think this is overkill and plan to replace it with another 100L4, expecting full load noise to drop a fair bit with that -- at least to my ears, the fans only become noisy around 5000 RPM, below that they're quiet enough.

I also have a D-1587 (65W TDP) in a CSE-300, with two 100L4's for the CPU. I added a cardboard air shroud, and full load with PUE-2 runs the fans at less than 4500RPM with temperature stable at 85C. Optimal ramps up the fans to 5500RPM and drops temps down to 70C.

I agree that Noctua's are not an option for anything beyond the Atom boards. I have a C3558 (SYS-E200-9A) as a pfSense router/firewall. I replaced the stock fan with 3 Noctua A4x20's (12V PWM version), and even though the CPU is only 16W TDP, it idles at around 50C with the fans @2500RPM. The only good thing is that they are silent even at full speed, but they can't really cool much.

If you use a larger chassis than CSE-300, you'll most likely want to add some sort of air shroud, since the fans will generally be further away from the heatsink.

With the E-2100/2200, the TDP's go up quite a bit to 80/95W and I'm not sure they're actually limited to that (my D-1587 never exceeds the rated 65W according to turbostat), on the desktop side there's a lot of reviews that show the processor can go well over TDP. I think they will be difficult to keep quiet in 1U if you run them flat out.