One box to replace them all...

[tWiitri]

Build’s Name: The Box
Operating System/ Storage Platform: ESXi
CPU: Dual Xeon L5630
Motherboard: HP ML350 G6
Chassis: HP ML350 G6
Drives: 6x Toshiba 2.5" 5.4k SATA in RAIDZ2, single Samsung 850EVO for datastorage
RAM: 192GB PC3L running at 1066MHz
Add-in Cards: LSI 9211-8i
Power Supply: 460w

I have been following the forums for a while and I just have to share my ongoing experience of consolidating lots of hardware/software into one box. I have always been careful not to "put all the eggs in one basket" since if something breaks I do not wan't everything else being down or affected. I mainly play around with VMware but also like to tinker with Hyper-V and KVM. For my storage I have been using FreeNAS. Main reasons for playing around with virtualization is to learn and hopefully make something useful out of it.

My old setup was as follows:

FreeNAS for backups and filestorage (photos, home movies)
AMD x3 445, 16GB ECC, Asus M5A78L-M LX3, LSI 9211-8i, 6x WD Red 1TB, Fractal Design Define Mini

Virtualization hosts:
2x Sun Fire x4100 m2, 2x Opteron 2384, 32GB DDR2 ECC 800MHz, 4Gbit Fibrechannel

VM-storage:
FreeNAS, HP DL380 G5, 2x Xeon 5650, 32GB FBDIMM, LSI 1068e, 6x 146GB 10k SAS, 4Gbit Fibrechannel

I also have 4 other MiniITX boxes running vCenter, ActiveDirectory or what ever I am playing with for the moment. I don't have any licences since I am only testing and usually reinstall within a month, so the trial versions have worked without issue. Now I am in a position where I won't be able to bring everything down and need a stable platform. I decided for the VMware Free ESXi hypervisor.

The main issue with my old gear is noise, power and heat. I have everything setup in a basement room with poor ventilation. Performance has not been an issue since I usually do not have any persistent VM's running. That has changed and I need now a replacement for my home-FreeNAS box since one of the disk's started throwing write errors.

I have basically no money to use, but I have quite a lot of useful stuff that I have gathered over the years and I recently was given what is to be "The box to replace them all": HP Proliant ML350 G6.

It had a pretty basic setup:
Single Xeon E5504, 16GB ECC (8x2GB), empty 8x SFF harddrive cage, single 460w PSU

Please stay tuned for the build.

 

[tWiitri]

Initial impressions of the ML350 G6:

• It's loud! Not as loud as the existing servers, but still...
• HP has all the firmware behind a paywall.
• Performance of the bundled E5504 was really poor.

Firmware SPP-bundle was acquired through some googling and I appreciate that people make these available to us who do not have a service agreement with HP. There are settings in the BIOS to adjust the fans for "Reduced Acoustics" but it was still too loud to have powered on 24/7. The fans use a propietary 5-pin connector and after som intense googling and measuring I was able to use normal 3-pin fans, without PWM-steering (since Hp seems to have some inverse way of doing it on their fans), with the help of a little adapter:

The first two pins from the top are Ground (looped together) followed by 12v and RPM signal. The last not connected pin is for the PWM-signal. I used a existing adapter with a 47Ohm resistor in place and just added the loop to the first pin. This way ILO did not complain and the fans connected would run at a fixed speed.

I decided to mount a Noctua 80mm fan directly to the CPU-heatsink. This way I hoped that a lower RPM was needed to keep it cool rather than having the 3 92mm case fans indirectly provide cooling.

I also made a deal with a local dealer to get a heatsink so I could mount an additional processor and also 6 new Toshiba OEM 1TB 2.5" 5400rpm harddrives for a total of approximately 90$US (converted from Swedish Krona).

Now I needed a CPU-upgrade. The most logical part for a low-power virtualizationserver would have been the Xeon L5640. 6 cores at 2.26GHz, 1333MHz memory speed, 60w TDP. But looking at the prices on ebay I settled for a pair of Xeon L5630 for just 20$US. 4 cores at 2.13Ghz, slower memory speed at 1066MHz but a even lower TDP at 40w. With 2 CPUS's installed ILO insisted on that at least 3 fans had to be connected so I added a Noctua 92mm case fan with a similar adapter. I actually have one more of those case fans but since the server came with only 3 of the 4 fans installed I ran out of those 5-pin fan connectors. I guess i could use a 4-pin fan connector but adding one more fan does not seem necessary.

I tested temperatures by installing a copy of Windows 10 Pro and running Prime95 for 12 hours. The processor core-temperatures were at most at 65 degrees celcius. But the chipset was burning up. The reduced airflow through the chassis was not sufficient to keep it cool enough. I exchanged the miniscule heatsink with at Thermalright cooler with heatpipes and also mounted a 120mm fan on a watercooling pump-mount at the bottom of the case to provide some airflow over the bottom half of the motherboard. This fan gets power from a Molex-connector. In the picture there is a AMD 7550 videocard that was used during testing. It has since been removed.

 

[tWiitri]

My initial plan was to run Ubuntu with ZFS and KVM/QEMU on the server and I got everything installed when the OCZ Vector SSD I had used suddenly died. The system hung and after a reboot the drive was not accessible any more. Fortunately OCZ/Toshiba have a great support and the drive is on the way to RMA.

After the drive failure I decided to go for ESXi since I find it less challenging to setup and not as time consuming. At first the plan was to use the onboard HP P410 controller and 4 old Samsung 64GB SSD's in RAID10 when the FBWC battery died. So I instead installed ESXi on a SD-card and use for the moment a single Samsung 850EVO 250Gb SSD as datastorage. The six 1TB drives are connected to a LSI 9211-8i which is passthrough to a FreeNAS VM (backups and photos, rarely accessed) and setup as a RAIDZ2.

I am thinking of either adding two SSD's to that VM and presenting them to ESXi through iSCSI but I am afraid there will be a lot of overhead. I will have to test that later on.

The last part missing from the build was RAM. 16GB was not going to last for long. Fortunately I came over 12 pieces of 16GB 1600MHz PC3L Reg DIMM for a mere 230$US. 192Gb RAM seems excessive considering the rather modest setup so far, but they were cheap. I am pretty sure that they are some sort of counterfeit RAM since the stickers vary in print and seem not well done. SPD says Micron and the stickers says Elpida. I have run memtest86 on them for hours and hours and checked the ILO-log for ECC errors, so far so good.

The server is really quiet now. I was amazed over how quiet the PSU fan is. I actually had ordered a few replacement fans since I anticipated that it would need replacing. The system draws 90w from a 230v outlet. Running Prime95 it drew 160w at full load.

I have a few questions for all of you who seem really experienced:

• How would you solve the datastorage on the server. RAID? Another zpool of SSD's?
• Would you recommend me to stick to ESXi or do you have any other recommendations? (the hosted VM's will mostly be Linux and Windows guests)
• Any other points that I have missed?

Thanks for reading!

 

[Esat]

How? How you solved 5 pin fan problem? Give us more detail please.. (pinout,schematic bla bla)
I will do same conf. as workstation for 3dsmax with quadro K600 or k2000

 

[pgh5278]

If you have two cpu the ml350 g6 it will expect 4 fans, you have two which is for one cpu, they are cheap on ebay. I have ml350 g6 and it was not loud when running..

 

[Esat]

If you have two cpu the ml350 g6 it will expect 4 fans, you have two which is for one cpu, they are cheap on ebay. I have ml350 g6 and it was not loud when running..

For now I have one cpu and three hp fans. Windows 10 pro and very loudy every single load / job even copy some files etc.
Already tried new hp fans.
Didn't change anything same loud
I just want to try tWiitri's way for loud problem.
At the moment I found this maybe helps my problem
I still google it
Anyway thanks for your answer/help
Ps: I from Turkey and we cant used ebay cause goverment issues.