Here is my experience so far in case it helps anyone else:
- Out-of-the-Box
- Server idles around 82W w/ 8x16GB RAM, 2x E5-2650v2, 2x 200GB DC S3710 SSDs booted into proxmox 7.1
- I needed to add a grub boot delay because when I boot to the ZFS RAID 1 mirror, it drops to initramfs and says it can't import rpool, but I import it manually and it comes right up. Adding in a delay of 10s in grub resolved the issue
- I'm getting ~49dB from the server @ 1m at idle. It's a full 76dB lulzfest at full bore.
- There are 6x Sunon PF40561B1-Q080-S99 system fans. They are 40x40x56mm 10-pin 18kRPM 27CFM 68dBA. Datasheet here
- Gotchas
- The RJ45 to the right of the VGA on the Intel S2600GZ mobo is SERIAL, not BMC/IPMI. If you plug in a network cable, you'll get spurious keyboard input if serial console redirection is enabled. IPMI is on NIC1
- The Intel S2600GZ motherboard has two SFF-8087 ports on it both connected to the on-board controller.
- Only port 0 is active out of the box
- Only SATA is supported out of the box
- The on-board controller can only support SATA 1.5/3Gbps or SAS 1/2 (3Gbps) (if enabled)
- The first two vertical pairs of drives are connected to the on-board controller port 0. The second two vertical pairs of drives are unconnected but would connect to port 1
- There is no second SFF-8087 cable connecting the controller to the SAS/SATA backplane out-of-the-box
- The vertical pair of drives all the way to the right are connected to the on-board 6Gbps SATA ports
- The backplane is 6Gbps maximum
- An RMM upgrade key is needed for remote KVM support (which itself, requires Java)
- Options
- To address #1, you need an upgrade key to activate port 1.
- To address #2, you need an upgrade key to enable SAS.
- To address #3, you need an IO module or an add-on PCIe card and riser. Be warned, if you get one of these IO modules, I'm not sure if you can flash them into IT mode and they don't support JBOD AFAICT, so they may be bad juju for ZFS/TrueNAS or potentially even Unraid. A PCIe riser board and an LSI 9207-8i/9211-8i or equivalent in IT mode is likely easier and cheaper in the end.
- To address #5, you need another cable. The straight SFF 8087 - 8087 #11 cables fit just fine into the backplane and also into my LSI 9207-8i. If you use an H200/PERC that has the SFF connectors on the top of the card and not facing the rear, then you'll need to put the card in the slot closest to the PSU. Cable length of 1m was a bit too long for the longest run, so the 0.8m cables are probably best
- There aren't any options to address #7 that I'm aware of
For 1 & 2 (Keys)
So if you want:
- Port 0, 4 drive SAS, you need RKSAS4 "Green".
- Port 0 & 1 8 drive SAS no-raid 5, you need RKSAS8 "Orange" (~$20 on eBay)
- Port 0 & 1 8 drive SATA no-raid 5, you need RKSATA8 "Blue" (~$15 on eBay)
- etc
SATA3/SAS2.1 IOModules
Firmware Updates
My servers had fairly dated ME and BMC firmware versions on them. Installing the
latest from Intel exposed some more options and did quiet the server down some (more on that in a moment).
Code:
System BIOS: 02.06.0006
ME Firmware: 02.01.07.328
BMC Firmware: 01.28.10603
FRUSDR: 1.14
In the BIOS, I have:
- "Power and Performance" as "Balanced Performance"
- "System Acoustic and Performance Configuration" as:
- Auto
- 300m or less (choose what is accurate to your location)
- Acoustic
- 0
- Enabled
Sensor Data Record (SDR) Editing
Even after the BIOS updates and edits, the server was still fairly loud at idle (~49dBA) and wouldn't start spooling up the fans until the CPU cores were hitting 75C, near the thermal limit. I also noticed that the fan speeds didn't seem right. I suspect that the Intel "Other Chassis" SDR profile doesn't actually match these fans and isn't a great fan curve. To address this, you can edit/flash a custom SDR (instructions below, my SDR is attached)
- Flash the system with the latest BIOS [1]
- Reboot the system
- Press F6 to get a boot menu
- Select `Internal EFI Shell`
- Press `ESC` to skip running `startup.nsh`
- Edit/copy the custom S2600GZ.SDR file onto the USB flash key you used to update the BIOS with*
- Plug-in the USB key to the Wiwynn
- Map the drive `map -r` in the EFI shell
- Switch to the drive (e.g. `fs2:` if your USB key mapped to that in the previous command)
- Run the FRU/SDR update script `UpdateFRUSDR.nsh`
- Select "Update only the SDR" by pressing `1`
- Select "Other Chassis" by pressing `3`
- Select "Slow Ramp" by pressing `1`
- System FAN 1A/B - 6A/B are all present, press `y` for each
- CPU 1 and CPU 2 fan are not present, nor is the Chassis Intrusion, press `n` for each
- Wait for flash to complete
- Exit
* You can perform all the file edits in the EFI Shell by switching to the drive and doing `edit S2600GZ.SDR` This is useful when you are tuning different value repeatedly so you don't have to keep shuttling the USB key
**NOTE** I have only edited the Other Chassis Slow Ramp profile. If you use another chassis or ramp profile, you'll get the defaults.
Be sure and run sensors on the device and watch your temps, especially under load (I ran mprime95 -t for about 25mins)
Also note, those "temps" in the fan profile are a bit misleading. Either the ME is following the VRM temp for the fan profile there Or (and more likely) it's using the combination of all the clamp values to calculate which index to use for fan speed. The key is those are not package/core temps in those values
Results
With 2x E5-2650v2 (95W) processors and the updated SDR, it ramps up to ~60% fan (9000/7200RPM) and holds the temps at 66-72C under AVX FFT prime95. 3300/2750RPMs is the lowest I can go. I suspect that 10% PWM is the lower limit for controlling the fans at
With 21C ambient:
- Idle : 36-45C, 3300/2750RPMs, 46dBA
- Max Load: 65-72C, 9000/7200RPMs, 61dBA
which is about 3dBA quieter under idle and 8dBA quieter under load. It would probably be closer to 55dBA under full load if the PSU fan wasn't so loud. good enough for now
[1]
Intel® Server Board S2600GZ/GL Firmware Update Package for Extensible Firmware Interface (EFI)
References:
[After a discussion with Andrew Su, this post has been updated to include information on how to select the temperature sensor that controls the fan speed.] I run a home server built around the Intel S1200V3RP motherboard. To minimize noise levels, it is housed inside a Fractal Design Define Mini...
sigg-iten.ch
Future
- PSUs are Delta DPS-650AB-6 B and have Delta FDB0412UHN-CF00 fans in them. Those are 40x40x28mm 3-pin 27.1cfm 2.11” H20 18.5K RPM 58.5dBA. They are pretty easy to remove and should be plenty of quieter options with good airflow
