Silent/Headless Workstation (X10DRi, 2xXeon E5, Phanteks Enthoo Pro)

[rasenmaeher]

Hello,

I've just finished my first build, I'll start with the specs and conclude with notes about my usage and part selection.

Operating System/ Storage Platform: Ubuntu 16.04
CPU: 2x Intel Xeon E5-2683v4 ES (2x16 Cores)
Motherboard: Supermicro X10DRi-o
Chassis: Phanteks Enthoo Pro (cheapest option with SSI-EEB/E-ATX support)
Drives:
- boot: SanDisk 960GB SATA
- secondary: 512GB Samsung SM951-NVMe SSD (MZVPV512HDGL)
RAM: 8x 16GB DDR4 REG ECC Crucial (CT16G4RFD424A)
Add-in Cards: Delock PCIe 1xM.2 NGFF NVMe (not bootable )
Power Supply: Corsair RMi Series RM850i (second CPU support by default)
Other Bits:
- CPU fans: 2x Noctua NH-U12DX i4

Usage Profile:
- Virtualization and cluster deployment experiments (DevOps) at home
- runs on demand, powered on/off by IPMI/LAN

It runs nicely, however I've still some issues:

1. Fans

- had to remove the Noctua low noise adapter (LNA) because FANs somehow reported sometimes 0 RPM which made the BMC trigger full fan speed.
- lowered IPMI fan thresholds and used IPMI raw commands to reduce minimum fan speeds
- CPU fans now run at 300-600RPM depending on the load, almost silent.
- I'm not fully satisfied with the case fan setup yet. They are 3pin fans connected to a fan controller (see last picture), then connected to a 4pin PWM port on the mainboard. Due to limitations of Supermicro and those fands, I cannot control them separately. (You can set speeds for 2 groups: CPU and System, not on a per-fan-basis.) While I manage to get them to 700RPM (number is provided by the rear fan, only), this leads to my problems:

2. Problems

1. PCH-temperature is rather high, starts easily to get in the range of 45-55°C. While the Intel C612 data sheet says maximum temperature is 92°C, the SuperMicro support tells me it is 60°C. I suspect this also starts spin up of the system fan group around 45°C already. SuperMicro support told me, that in server environments there is usually enough air flow to cool the PCH and they have other boards for workstation usage - right, but I wanted IPMI which they seem not to provide with their workstation boards. I've already removed the unused drive cage to increase air flow from the front fan over the PCH but this didn't make any difference. So I'm probably have to mount a small fan on top of the PCH heatsink - I'm open for any recommendation to solve this issue.

2. NVMe

Looks like the Supermicro AMI Bios is not able to boot from NVMe without an option ROM, as it is provided by "native" PCIe NVMe-cards, for example by Intel. The M2/PCIe-adapter I'm using seems not to have an option ROM and the board itself does not support NVMe in its EFI targets. Yes, I've set everything to EFI in the BIOS, but still does not recognize the NVMe SSD as boot-option.

However the NVMe works fine once the system is booted. I'm now using a legacy SATA SSD to boot, using the NVMe SSD to store VM data and docker images/layers.

Usage:

I'm a consultant for DevOps/Infra solutions and I'm using this machine to run 100s of docker containers and kvm-based VMs to evaluate things like Kubernetes/OpenShift setups. While I can work from home most of the time, my upstream bandwidth is rather limited to 12MBit/s so renting a box at some cheap server provider (or AWS) wasn't an option. Also I don't want to have customer data been deployed on test-VMs/machines somewhere in the cloud outside of the customer's reach. The low per-core performance (~2 GHz only) is acceptable for my usage. Using the nice IPMI and KVM functionality, I can switch the box on/off from the shell on my main computer (Macbook Pro Retina, unfortunately max 16GB RAM…) and then use GBit-LAN or 802.11ac to interact. This was the main reason not to use the workstation boards like X10DAX mobo, which provide more features like Hyperspeed, SLI support, audio - but no IPMI afaik.

Thanks!

Thanks a lot for all the resources on ServeTheHome about dealing with Supermicro fan settings (and ES processors).

 

[Bill1950]

Ee gads. What a multi-threaded monster.

Very nice clean build.

Have you considered using a manually controlled variable fan speed controller or two such as the ones supplied by Zalman with some of their CPU coolers? Use the controllers on the case fans. With that kind of set-up you could manually set the speed of a fan or two to lower the noise level and allow other case fans to react to mb temps. If the temps are regularly exceeding what you consider the temp comfort level, manually dial up the fan speeds until the happy place is found.

 

[Mkmdad]

Hello,

I've just finished my first build, I'll start with the specs and conclude with notes about my usage and part selection.

Operating System/ Storage Platform: Ubuntu 16.04
CPU: 2x Intel Xeon E5-2683v4 ES (2x16 Cores)
Motherboard: Supermicro X10DRi-o
Chassis: Phanteks Enthoo Pro (cheapest option with SSI-EEB/E-ATX support)
Drives:
- boot: SanDisk 960GB SATA
- secondary: 512GB Samsung SM951-NVMe SSD (MZVPV512HDGL)
RAM: 8x 16GB DDR4 REG ECC Crucial (CT16G4RFD424A)
Add-in Cards: Delock PCIe 1xM.2 NGFF NVMe (not bootable )
Power Supply: Corsair RMi Series RM850i (second CPU support by default)
Other Bits:
- CPU fans: 2x Noctua NH-U12DX i4

Usage Profile:
- Virtualization and cluster deployment experiments (DevOps) at home
- runs on demand, powered on/off by IPMI/LAN

It runs nicely, however I've still some issues:

1. Fans

- had to remove the Noctua low noise adapter (LNA) because FANs somehow reported sometimes 0 RPM which made the BMC trigger full fan speed.
- lowered IPMI fan thresholds and used IPMI raw commands to reduce minimum fan speeds
- CPU fans now run at 300-600RPM depending on the load, almost silent.
- I'm not fully satisfied with the case fan setup yet. They are 3pin fans connected to a fan controller (see last picture), then connected to a 4pin PWM port on the mainboard. Due to limitations of Supermicro and those fands, I cannot control them separately. (You can set speeds for 2 groups: CPU and System, not on a per-fan-basis.) While I manage to get them to 700RPM (number is provided by the rear fan, only), this leads to my problems:

2. Problems

1. PCH-temperature is rather high, starts easily to get in the range of 45-55°C. While the Intel C612 data sheet says maximum temperature is 92°C, the SuperMicro support tells me it is 60°C. I suspect this also starts spin up of the system fan group around 45°C already. SuperMicro support told me, that in server environments there is usually enough air flow to cool the PCH and they have other boards for workstation usage - right, but I wanted IPMI which they seem not to provide with their workstation boards. I've already removed the unused drive cage to increase air flow from the front fan over the PCH but this didn't make any difference. So I'm probably have to mount a small fan on top of the PCH heatsink - I'm open for any recommendation to solve this issue.

2. NVMe

Looks like the Supermicro AMI Bios is not able to boot from NVMe without an option ROM, as it is provided by "native" PCIe NVMe-cards, for example by Intel. The M2/PCIe-adapter I'm using seems not to have an option ROM and the board itself does not support NVMe in its EFI targets. Yes, I've set everything to EFI in the BIOS, but still does not recognize the NVMe SSD as boot-option.

However the NVMe works fine once the system is booted. I'm now using a legacy SATA SSD to boot, using the NVMe SSD to store VM data and docker images/layers.

Usage:

I'm a consultant for DevOps/Infra solutions and I'm using this machine to run 100s of docker containers and kvm-based VMs to evaluate things like Kubernetes/OpenShift setups. While I can work from home most of the time, my upstream bandwidth is rather limited to 12MBit/s so renting a box at some cheap server provider (or AWS) wasn't an option. Also I don't want to have customer data been deployed on test-VMs/machines somewhere in the cloud outside of the customer's reach. The low per-core performance (~2 GHz only) is acceptable for my usage. Using the nice IPMI and KVM functionality, I can switch the box on/off from the shell on my main computer (Macbook Pro Retina, unfortunately max 16GB RAM…) and then use GBit-LAN or 802.11ac to interact. This was the main reason not to use the workstation boards like X10DAX mobo, which provide more features like Hyperspeed, SLI support, audio - but no IPMI afaik.

Thanks!

Thanks a lot for all the resources on ServeTheHome about dealing with Supermicro fan settings (and ES processors).

Hi there
I recently built a system with supermicro x10dal-I, 2x e5-2660 v3 and here is what I will share with you.
First, I am booting off a Samsung 951 nvme in an adapter bracket. My bios is set to UEFI before installing OS. Windows 10.
As for Temps, I see possible improvements. To create better flow your cpu cooler fans should point up not down , the way you have it set up they are fighting the fan in the front of the case.
Second, and what would make a huge difference is if you add fans at the top side of the case. Your case can take up to 3 140mm fans at the top.
Using fans at the top of my case and pointing the cpu fans up I am running at 38-40c. And under extreme load never went over 54c.
And for the record check out the Intel cpu coolers and notice that the fins are right on top of the cpu unlike the ones you have.
In my opinion the intel coolers are better and also cheaper.
Good luck.

 

[rasenmaeher]

Ee gads. What a multi-threaded monster.
Have you considered using a manually controlled variable fan speed controller or two such as the ones supplied by Zalman with some of their CPU coolers? Use the controllers on the case fans.

The board is able to control the fans based on the heat (well, the combo of the two case fans). While I can manually set the low-speed to a very slow and silent level, the PCH (mainboard chipset) starts to get way above 40°C and this seems to trigger the mainboard/IPMI to increase the fan speed to a somewhat noisier level. I guess If I put hardware in place to reduce the fan speeds, the board will even further increase the fan speed to keep the temperature of the PCH constant.

The used Phanteks case has 2 fans (1x200mm front, 1x140mm rear top) pre-installed but both are far away from the PCH (that's the guy with the heatsink above the PSU). meh.

 

[rasenmaeher]

Hi there
I recently built a system with supermicro x10dal-I, 2x e5-2660 v3 and here is what I will share with you.
First, I am booting off a Samsung 951 nvme in an adapter bracket. My bios is set to UEFI before installing OS. Windows 10.

May I ask which adapter bracket you're using? Supermicro support told me that many users have problems with adapters by "DeLock" so I'm ready to try another vendor.

As for Temps, I see possible improvements. To create better flow your cpu cooler fans should point up not down , the way you have it set up they are fighting the fan in the front of the case.

CPU temp is 31-52°C so the cooling works very good IMHO. The fans point up, they push the air through the heatsink to the top of the case.

So the only problem is the PCH temperature (the chip with the heatsink just above the PSU at the bottom).

 

[Mkmdad]

May I ask which adapter bracket you're using? Supermicro support told me that many users have problems with adapters by "DeLock" so I'm ready to try another vendor.



CPU temp is 31-52°C so the cooling works very good IMHO. The fans point up, they push the air through the heatsink to the top of the case.

So the only problem is the PCH temperature (the chip with the heatsink just above the PSU at the bottom).

Lycom DT-120 is the bracket that I am using.

 

[marcus bennett]

Hi from Marcus.

I am also using a X10DRi in fact the X10DRi-T variant. I cannot boot from NVMe disk either. I have tried everything, and I mean everything. The September 2016 BIOS is installed. There are options to select EFI at the PCIe Slot level and to select UEFI boot, but nothing. I realised that the proof of non boot is that when starting into the EFI shell that the map command does not show any PCIe disk. I would argue that it should even before any OS is installed to that PCIe NVMe disk. It does not. I raised a support called with Supermicro and I am almost into 2 months but no progress. I got to the stage where I tested all my components: NVMe disk, riser card and so forth in other systems. Just as a precaution and they all work fine. So I await to hear from Supermicro!!

In my build I am using 2 x Corsair H90 coolers which I think are so much more practical and prettier than Fan based cooling (you have to buy a separate CPU mounting bracket though).

 

[marcus bennett]

Might I also add forNVMe booting. The PCIe adapters are basically passive, well mine is anyway. I have tried 3 of them, they are simply a bracket with circuit traces down to the Motherboard. There would seem to be 0 scope for any errors. And to confirm the disk is accessible if the machine is booted and installed from say SATA medium.

Supermicro do a PCIe riser card
AOC-SLG3-2M2
Super Micro Computer, Inc. - Products | Accessories | Add-on Cards | AOC-SLG3-2M2

But Supermicro said its not officially supported with this motherboard. (That card does bifurcation so you can split the PCIe lane up and run say 2 PCIe disks. Neat. However Supermicro support wrongly told me IMHO that Bifurcation is not supported on this motherboard. Well it is in the BIOS and I've tested out all the options that I can see and it seems to work perfectly to me)

 

[pricklypunter]

Nice clean tidy build

 

[SycoPath]

Might I also add forNVMe booting. The PCIe adapters are basically passive, well mine is anyway. I have tried 3 of them, they are simply a bracket with circuit traces down to the Motherboard. There would seem to be 0 scope for any errors. And to confirm the disk is accessible if the machine is booted and installed from say SATA medium.

Supermicro do a PCIe riser card
AOC-SLG3-2M2
Super Micro Computer, Inc. - Products | Accessories | Add-on Cards | AOC-SLG3-2M2

But Supermicro said its not officially supported with this motherboard. (That card does bifurcation so you can split the PCIe lane up and run say 2 PCIe disks. Neat. However Supermicro support wrongly told me IMHO that Bifurcation is not supported on this motherboard. Well it is in the BIOS and I've tested out all the options that I can see and it seems to work perfectly to me)

If they have tested it and it doesn't work, they will outright tell you can't do it. "Not Supported" is a very carefully chosen phraise. "Not Supported" means we never tested that configuration so we don't know if it works properly or not, so we recommend against it because that's what our lawyers recommend.

 

[Trotador]

You can use a 40x40 mm fan to cool the motherboard chipset attached directly to the heatsink

40x40x10mm fan | eBay

If needed, to speed down and minimise noise, a typical resistor bridge will make it.

I've done in older builds but not recently since those chipsets have normally pretty high heat ratings