FINAL Bachelor Build - Xeon D vSAN Cluster

[msvirtualguy]

This is a pretty slick setup and very similar to what i'm moving to over the summer albeit instead of VSAN i'll be running Nutanix CE. Please keep us posted on Heat/Noise as you fire up these beasts. It would be very interesting to see since I have some WAF issues right now with my gear.

 

[IamSpartacus]

This is a pretty slick setup and very similar to what i'm moving to over the summer albeit instead of VSAN i'll be running Nutanix CE. Please keep us posted on Heat/Noise as you fire up these beasts. It would be very interesting to see since I have some WAF issues right now with my gear.

Will do @virtualfng as heat and noise is a major factor for me.

 

[whitey]

Shhhhhhhh...ya hear that...that's the sound of us jealously cursing you under our breath :-D

j/k, VERY nice setup bud, get it all out of your system while you can!

I want one but don't NEED one, the 3-node 2670 cluster is killin' it over here but I bet my 4 amp usage is gonna double yours w/ your 3-nodes lit up. I am REALLY interested to see/hear results back on pwr usage, ya got a meter or PDU that will show it?

 

[IamSpartacus]

Shhhhhhhh...ya hear that...that's the sound of us jealously cursing you under our breath :-D

j/k, VERY nice setup bud, get it all out of your system while you can!

I want one but don't NEED one, the 3-node 2670 cluster is killin' it over here but I bet my 4 amp usage is gonna double yours w/ your 3-nodes lit up. I am REALLY interested to see/hear results back on pwr usage, ya got a meter or PDU that will show it?

Yup have an APC 7900 PDU on it's way so once I get the whole rack configured I'll post my power usage.

 

[whitey]

That'll do!

 

[IamSpartacus]

So some bad news to start things off...It appears the 2U cases I got are not going to be sufficient to cool

these D-1537's. With the two 80mm fans positioned where they're supposed to be in the front of the case the CPU temps eventually get up over 80C. Even when I position my fans right in front of the heatisink and turn them on 100% my CPU sits at 53-54C after POST and sitting at the "please select boot device" screen while all the other components appear fine (see below). I'm wondering if the problem is the case's lack of exhaust thus the hot air has no where to go.

I'm a little perplexed by this because even though my D-1540 has an active HSF the intake fans are nowhere near as powerful as the ones in the D-1537 case. However the D-1540 is in a 3U case with dual 80mm exhaust fans so maybe that is the difference.

 

[Quasduco]

I'd cut up a Manila file folder into a duct and see how that does.

 

[IamSpartacus]

I'd cut up a Manila file folder into a duct and see how that does.

OK so ducting does seem to do the job. With a duct from one of the front 80mm fans to the CPU the temps don't go above 76 when I stress all 8 cores at 100%. Then at idle with the fans turned down temps are in the low to mid 40's.

The question is, how do I configure the fans to automatically speed up when CPU temps reach a certain number because at this point they don't. When I started the stress test the fans did not initially speed up I had to manually speed them up in IPMI.

 

[miraculix]

OK so ducting does seem to do the job. With a duct from one of the front 80mm fans to the CPU the temps don't go above 76 when I stress all 8 cores at 100%. Then at idle with the fans turned down temps are in the low to mid 40's.

The question is, how do I configure the fans to automatically speed up when CPU temps reach a certain number because at this point they don't. When I started the stress test the fans did not initially speed up I had to manually speed them up in IPMI.

• The 80mm fan you're ducting... if it's currently plugged into one of the 3 chassis fan PWM headers, move it over to the dedicated CPU fan PWM header
• The 80mm SM fans you mentioned in your HW list are 4-pin PWM, so you can easily change all fan speeds and temp thresholds in the BIOS
• Ducting is probably fine, but you may want to consider adding a dedicated 60mm CPU fan assuming you have enough clearance. Noctua has great 60mm CPU fans but securing them may be an issue. If the heatsinks were short (doesn't look like they are in the pics) I'd recommend high temp epoxy to secure the "feet" of the Noctua fan silicone anchors to the tops of the HS screws... but plenty of other options given enough time and thought.

 

[IamSpartacus]

• The 80mm fan you're ducting... if it's currently plugged into one of the 3 chassis fan PWM headers, move it over to the dedicated CPU fan PWM header
• The 80mm SM fans you mentioned in your HW list are 4-pin PWM, so you can easily change all fan speeds and temp thresholds in the BIOS
• Ducting is probably fine, but you may want to consider adding a dedicated 60mm CPU fan assuming you have enough clearance. Noctua has great 60mm CPU fans but securing them may be an issue. If the heatsinks were short (doesn't look like they are in the pics) I'd recommend high temp epoxy to secure the "feet" of the Noctua fan silicone anchors to the tops of the HS screws... but plenty of other options given enough time and thought.

Thanks for the input. The problem is there is no manual for this board yet since it's so new so I'm not sure which fan header is for the CPU. They are just labled FAN1, FAN2, FAN3, FAN4, FANA, FANB.

There is room to secure a fan to the heatsink but yes I will need to figure out a way to secure them. I will look into that.

 

[vsxi-13]

Thanks for the input. The problem is there is no manual for this board yet since it's so new so I'm not sure which fan header is for the CPU. They are just labled FAN1, FAN2, FAN3, FAN4, FANA, FANB.

There is room to secure a fan to the heatsink but yes I will need to figure out a way to secure them. I will look into that.

I just recently put together a new CPU/Supermicro Mobo that did not have a dedicated CPU fan power port. In the instructions it it stated the following in the connectors section:

Connector |||||||||||||||||||||||||||| Description
Fan1-Fan4, FanA ||||||||||||||||||| System/CPU Fan Headers

Based on how they put together that section, it made me think that FanA was for the CPU fan header.

 

[IamSpartacus]

I just recently put together a new CPU/Supermicro Mobo that did not have a dedicated CPU fan power port. In the instructions it it stated the following in the connectors section:

Connector |||||||||||||||||||||||||||| Description
Fan1-Fan4, FanA ||||||||||||||||||| System/CPU Fan Headers

Based on how they put together that section, it made me think that FanA was for the CPU fan header.

Thanks for this info I'll test this and see if that's the case.


On an unrelated note, I turned my UPS on for the first time today and holy hell is the system fan loud. I was not aware that the online series of these UPS' had to have the system fan running 100% all the time. This is just not a workable unit for my home office so I'm going to have to exchange it for a different model unfortunately.

 

[msvirtualguy]

Watching this thread closely.....

 

[PigLover]

All SM passive CPU coolers are designed to operate in a chassis with active airflow over the HSF. In general, they are designed to operate in their 1U-4U chassis where the mid-plane fans are positioned to blow over the CPU. In most of their chassis they also include a plastic shroud to ensure that the airflow of at least 2-3 of the midplane fans hit the CPU HSF directly.

They are not designed to be passive - they just don't have their fan attached.

In your case the duct is probably the best solution. A dedicated fan would be OK too, but it should be set up to blow ACROSS the heatsink - front to back - not straight down into it like an active HSF might be built.

As for which fan header is CPU - it probably doesn't matter. Any of fan1-fan4 should be OK. Set the fan speed to "optimal" in IPMI and it should run them as slowly as possible and spin them all up together when things heat up.

In most SM boards, fanA/B are generally for the "second heat zone" and might not vary themselves together with the other fans. Depending on the case they designed to board to pair with "second heat zone" might refer to exhasut fans or (more likely in this case) fans targeted at add-on boards like HBAs.

 

[miraculix]

Thanks for the input. The problem is there is no manual for this board yet since it's so new so I'm not sure which fan header is for the CPU. They are just labled FAN1, FAN2, FAN3, FAN4, FANA, FANB.

There is room to secure a fan to the heatsink but yes I will need to figure out a way to secure them. I will look into that.

It may be irrelevant to FlexATX X10SDVs but FYI my 8C-TLN4F based 1U 5018D-FNT4s came with one (ducted) fan connected to the FAN1 header, right next to the 24-pin ATX power plug.

EDIT: @PigLover and @Quasduco make a great point about ducting being good enough here, especially with the more defined front-to-back airflow in your (literal) case. As long as you're happy enough with your handiwork so the OCD doesn't kick in!

Full disclosure... the one Xeon-D system I added a CPU fan is a passive/fanless X10SDV-4C-TLN2F, installed in a Silverstone DS380. My other 3 are the 1U systems mentioned above and they seem fine with respect to temps.

 

[Ramos]

Great pictures! You should have seen them pop into my mail box at 1920x2160.

Sorry to hear the cooling is an issue.

Perhaps a stupid question, but thought of water cooling to just "put a pad" (e.g. All-in-one) on the CPUs and then running the water out to where ever there is room for a radiator ? ... Would save a lot of hassle I think. But of cause water has its risks.

 

[IamSpartacus]

All SM passive CPU coolers are designed to operate in a chassis with active airflow over the HSF. In general, they are designed to operate in their 1U-4U chassis where the mid-plane fans are positioned to blow over the CPU. In most of their chassis they also include a plastic shroud to ensure that the airflow of at least 2-3 of the midplane fans hit the CPU HSF directly.

They are not designed to be passive - they just don't have their fan attached.

In your case the duct is probably the best solution. A dedicated fan would be OK too, but it should be set up to blow ACROSS the heatsink - front to back - not straight down into it like an active HSF might be built.

As for which fan header is CPU - it probably doesn't matter. Any of fan1-fan4 should be OK. Set the fan speed to "optimal" in IPMI and it should run them as slowly as possible and spin them all up together when things heat up.

In most SM boards, fanA/B are generally for the "second heat zone" and might not vary themselves together with the other fans. Depending on the case they designed to board to pair with "second heat zone" might refer to exhasut fans or (more likely in this case) fans targeted at add-on boards like HBAs.

I'm going to use a duct for one of the 80mm intake fans and blow it directly at the CPU and then have a 60mm noctua directly in front of the heatsink as well. I have some of this epoxy do you happen to know if this can be used to safely secure things (in this case the 60mm fan) to a motherboard?

It may be irrelevant to FlexATX X10SDVs but FYI my 8C-TLN4F based 1U 5018D-FNT4s came with one (ducted) fan connected to the FAN1 header, right next to the 24-pin ATX power plug.

EDIT: @PigLover and @Quasduco make a great point about ducting being good enough here, especially with the more defined front-to-back airflow in your (literal) case. As long as you're happy enough with your handiwork so the OCD doesn't kick in!

Full disclosure... the one Xeon-D system I added a CPU fan is a passive/fanless X10SDV-4C-TLN2F, installed in a Silverstone DS380. My other 3 are the 1U systems mentioned above and they seem fine with respect to temps.

Thanks for this info. You were right FAN1 appears to be the correct fan header to use.

Great pictures! You should have seen them pop into my mail box at 1920x2160.

Sorry to hear the cooling is an issue.

Perhaps a stupid question, but thought of water cooling to just "put a pad" (e.g. All-in-one) on the CPUs and then running the water out to where ever there is room for a radiator ? ... Would save a lot of hassle I think. But of cause water has its risks.

In my hayday I'd certainly consider watercooling this setup but the big thing that turns me off from watercooling now is maintenance. Also with rack servers I want to quickly and easily be able to pull the servers out of the rack for maintenance/upgrades which would be a real PITA with watercooling.

 

[Quasduco]

Just me, I'd never epoxy anything to a motherboard - too delicate...

 

[IamSpartacus]

Just me, I'd never epoxy anything to a motherboard - too delicate...

What about the heatsink?

 

[Quasduco]

What about the heatsink?

Heat sinks, if glued, are epoxied to processors, which is expected and designed for.