One box to replace them all...

Discussion in 'DIY Server and Workstation Builds' started by tWiitri, Jul 26, 2016.

  1. tWiitri

    tWiitri New Member

    Joined:
    Jun 19, 2016
    Messages:
    6
    Likes Received:
    3
    Build’s Name: The Box
    Operating System/ Storage Platform: ESXi
    CPU: Dual Xeon L5630
    Motherboard: HP ML350 G6
    Chassis: HP ML350 G6
    Drives: 6x Toshiba 2.5" 5.4k SATA in RAIDZ2, single Samsung 850EVO for datastorage
    RAM: 192GB PC3L running at 1066MHz
    Add-in Cards: LSI 9211-8i
    Power Supply: 460w

    I have been following the forums for a while and I just have to share my ongoing experience of consolidating lots of hardware/software into one box. I have always been careful not to "put all the eggs in one basket" since if something breaks I do not wan't everything else being down or affected. I mainly play around with VMware but also like to tinker with Hyper-V and KVM. For my storage I have been using FreeNAS. Main reasons for playing around with virtualization is to learn and hopefully make something useful out of it.

    My old setup was as follows:

    FreeNAS for backups and filestorage (photos, home movies)
    AMD x3 445, 16GB ECC, Asus M5A78L-M LX3, LSI 9211-8i, 6x WD Red 1TB, Fractal Design Define Mini

    [​IMG]

    Virtualization hosts:
    2x Sun Fire x4100 m2, 2x Opteron 2384, 32GB DDR2 ECC 800MHz, 4Gbit Fibrechannel

    VM-storage:
    FreeNAS, HP DL380 G5, 2x Xeon 5650, 32GB FBDIMM, LSI 1068e, 6x 146GB 10k SAS, 4Gbit Fibrechannel

    [​IMG]

    I also have 4 other MiniITX boxes running vCenter, ActiveDirectory or what ever I am playing with for the moment. I don't have any licences since I am only testing and usually reinstall within a month, so the trial versions have worked without issue. Now I am in a position where I won't be able to bring everything down and need a stable platform. I decided for the VMware Free ESXi hypervisor.

    The main issue with my old gear is noise, power and heat. I have everything setup in a basement room with poor ventilation. Performance has not been an issue since I usually do not have any persistent VM's running. That has changed and I need now a replacement for my home-FreeNAS box since one of the disk's started throwing write errors.

    I have basically no money to use, but I have quite a lot of useful stuff that I have gathered over the years and I recently was given what is to be "The box to replace them all": HP Proliant ML350 G6.

    It had a pretty basic setup:
    Single Xeon E5504, 16GB ECC (8x2GB), empty 8x SFF harddrive cage, single 460w PSU

    [​IMG]

    Please stay tuned for the build. :)
     
    #1
    Last edited: Jul 27, 2016
  2. tWiitri

    tWiitri New Member

    Joined:
    Jun 19, 2016
    Messages:
    6
    Likes Received:
    3
    Initial impressions of the ML350 G6:
    • It's loud! Not as loud as the existing servers, but still...
    • HP has all the firmware behind a paywall.
    • Performance of the bundled E5504 was really poor.
    Firmware SPP-bundle was acquired through some googling and I appreciate that people make these available to us who do not have a service agreement with HP. There are settings in the BIOS to adjust the fans for "Reduced Acoustics" but it was still too loud to have powered on 24/7. The fans use a propietary 5-pin connector and after som intense googling and measuring I was able to use normal 3-pin fans, without PWM-steering (since Hp seems to have some inverse way of doing it on their fans), with the help of a little adapter:

    [​IMG]

    The first two pins from the top are Ground (looped together) followed by 12v and RPM signal. The last not connected pin is for the PWM-signal. I used a existing adapter with a 47Ohm resistor in place and just added the loop to the first pin. This way ILO did not complain and the fans connected would run at a fixed speed.

    I decided to mount a Noctua 80mm fan directly to the CPU-heatsink. This way I hoped that a lower RPM was needed to keep it cool rather than having the 3 92mm case fans indirectly provide cooling.

    [​IMG]

    I also made a deal with a local dealer to get a heatsink so I could mount an additional processor and also 6 new Toshiba OEM 1TB 2.5" 5400rpm harddrives for a total of approximately 90$US (converted from Swedish Krona).

    Now I needed a CPU-upgrade. The most logical part for a low-power virtualizationserver would have been the Xeon L5640. 6 cores at 2.26GHz, 1333MHz memory speed, 60w TDP. But looking at the prices on ebay I settled for a pair of Xeon L5630 for just 20$US. 4 cores at 2.13Ghz, slower memory speed at 1066MHz but a even lower TDP at 40w. With 2 CPUS's installed ILO insisted on that at least 3 fans had to be connected so I added a Noctua 92mm case fan with a similar adapter. I actually have one more of those case fans but since the server came with only 3 of the 4 fans installed I ran out of those 5-pin fan connectors. I guess i could use a 4-pin fan connector but adding one more fan does not seem necessary.

    I tested temperatures by installing a copy of Windows 10 Pro and running Prime95 for 12 hours. The processor core-temperatures were at most at 65 degrees celcius. But the chipset was burning up. The reduced airflow through the chassis was not sufficient to keep it cool enough. I exchanged the miniscule heatsink with at Thermalright cooler with heatpipes and also mounted a 120mm fan on a watercooling pump-mount at the bottom of the case to provide some airflow over the bottom half of the motherboard. This fan gets power from a Molex-connector. In the picture there is a AMD 7550 videocard that was used during testing. It has since been removed.

    [​IMG]
     
    #2
    Last edited: Jul 27, 2016
  3. tWiitri

    tWiitri New Member

    Joined:
    Jun 19, 2016
    Messages:
    6
    Likes Received:
    3
    My initial plan was to run Ubuntu with ZFS and KVM/QEMU on the server and I got everything installed when the OCZ Vector SSD I had used suddenly died. The system hung and after a reboot the drive was not accessible any more. Fortunately OCZ/Toshiba have a great support and the drive is on the way to RMA.

    After the drive failure I decided to go for ESXi since I find it less challenging to setup and not as time consuming. At first the plan was to use the onboard HP P410 controller and 4 old Samsung 64GB SSD's in RAID10 when the FBWC battery died. So I instead installed ESXi on a SD-card and use for the moment a single Samsung 850EVO 250Gb SSD as datastorage. The six 1TB drives are connected to a LSI 9211-8i which is passthrough to a FreeNAS VM (backups and photos, rarely accessed) and setup as a RAIDZ2.

    I am thinking of either adding two SSD's to that VM and presenting them to ESXi through iSCSI but I am afraid there will be a lot of overhead. I will have to test that later on.

    The last part missing from the build was RAM. 16GB was not going to last for long. Fortunately I came over 12 pieces of 16GB 1600MHz PC3L Reg DIMM for a mere 230$US. 192Gb RAM seems excessive considering the rather modest setup so far, but they were cheap. I am pretty sure that they are some sort of counterfeit RAM since the stickers vary in print and seem not well done. SPD says Micron and the stickers says Elpida. I have run memtest86 on them for hours and hours and checked the ILO-log for ECC errors, so far so good.

    [​IMG]

    The server is really quiet now. I was amazed over how quiet the PSU fan is. I actually had ordered a few replacement fans since I anticipated that it would need replacing. The system draws 90w from a 230v outlet. Running Prime95 it drew 160w at full load.

    I have a few questions for all of you who seem really experienced:
    • How would you solve the datastorage on the server. RAID? Another zpool of SSD's?
    • Would you recommend me to stick to ESXi or do you have any other recommendations? (the hosted VM's will mostly be Linux and Windows guests)
    • Any other points that I have missed?
    Thanks for reading!
     
    #3
    Last edited: Jul 27, 2016
    Esat, distracted247 and Kristian like this.
  4. Esat

    Esat New Member

    Joined:
    Dec 26, 2016
    Messages:
    2
    Likes Received:
    0
    How? How you solved 5 pin fan problem? Give us more detail please.. (pinout,schematic bla bla)
    I will do same conf. as workstation for 3dsmax with quadro K600 or k2000
     
    #4
  5. pgh5278

    pgh5278 Active Member

    Joined:
    Oct 25, 2012
    Messages:
    473
    Likes Received:
    124
    If you have two cpu the ml350 g6 it will expect 4 fans, you have two which is for one cpu, they are cheap on ebay. I have ml350 g6 and it was not loud when running..
     
    #5
  6. Esat

    Esat New Member

    Joined:
    Dec 26, 2016
    Messages:
    2
    Likes Received:
    0
    For now I have one cpu and three hp fans. Windows 10 pro and very loudy every single load / job even copy some files etc.
    Already tried new hp fans.
    Didn't change anything same loud
    I just want to try tWiitri's way for loud problem.
    At the moment I found this maybe helps my problem
    I still google it
    Anyway thanks for your answer/help
    Ps: I from Turkey and we cant used ebay cause goverment issues.
     
    #6
Similar Threads: replace
Forum Title Date
DIY Server and Workstation Builds ICY DOCK Fan Replacement Nov 14, 2018
DIY Server and Workstation Builds Replacement build advice Jul 29, 2018
DIY Server and Workstation Builds Quiet and efficient N54L replacement Jun 10, 2018
DIY Server and Workstation Builds After some advice on first FreeNAS build to replace current setup Sep 16, 2017
DIY Server and Workstation Builds ASRock J4205 Low Power Pi replacement Aug 20, 2017

Share This Page