2018 - Upgrading the toys

Discussion in 'DIY Server and Workstation Builds' started by nry, Apr 9, 2018.

  1. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    Got the itch to upgrade various things at the moment, all while renovating a whole house. Turning into a hectic year!

    I have come a long way in recent years, see my original build log at https://forums.servethehome.com/ind...ld-vsphere-cluster-and-network-overhaul.1712/ where I probably bought items without too much thought into how they will be used. Will try not continue this but given I probably have a spending problem, I make no promises.

    Recently posted a rough outline for upgrade options here https://forums.servethehome.com/index.php?threads/homelab-update-options-advise.18954/

    This thread probably won't be on the same level as some of the incredible setups posted here, but still hope some find it at least remotely interesting :rolleyes:

    Current setup
    Rack

    • APC UPS - SMT1000RMI2U
    • APC PDU ZeroU
    • Dell PowerConnect 5525P
    • Dell PowerConnect 5524 (stacked with above)
    • MikroTik CRS317-1G-16S+RM - not used currently, this will be the core switch
    • Main Server running ESXi 6 - runs 24/7
      • Supermicro X9SRL-F
      • Xeon E5-1620v2
      • 128GB ECC DDR3
      • 2x M1015 HBA
      • Samsung Evo 840 120GB boot drive
      • Samsung SM961 1TB NVMe - primary datastore
      • Samsung Evo 850 1TB - local backup datastore
      • WD 6TB Red - Media storage due to storage constraints on ZFS pool
      • 12x Hitachi/HGST 3TB 5k3000 drives (might be 1 or 2 7K3000 drives mixed in)
      • Intel X520-DA2 SFP+ NIC
    • Secondary server running ESXi 6 or what ever OS I need to experiment with
      • Asus Z9PE-D8 WS (with IPMI chip)
      • 2x Xeon E5-2660
      • 128GB ECC DDR3
      • Mixed old SSDs for testing
      • 3-4 Hitachi 3TB 7K3000 drives
      • Asus HD5450 (used to passthrough to VM)
      • Intel X520-DA2 SFP+ NIC
    Desk/Workstation
    • Desktop running Mac OS
      • Gigabyte GA-Z97X-UD3H
      • i7 4790K
      • 32GB Corsair DDR3
      • Samsung SM951 512GB
      • Samsung 940 Pro 256 GB (Windows boot drive)
      • Corsair CX650
      • GTX 980
      • Myricom 10G-PCIE2-8B2-2S (dual port SFP+ card)
      • Generic bluetooth/WiFi card for Mac
    • 3x Dell P2715Q Monitors
    • Apple Keyboard/Trackpad
    • Razor Mamba Mouse
    • Audiolab 8000P Amp
    • Audolab M-DAC
    • B&W 685 S2 speakers
    • 15" MacBook Pro 2014 (i7, 16GB RAM, 512GB SSD)
      • Sonnect Echo Express with Myricom 10G-PCIE-8B-S (1 port SFP+ card)
    Misc
    • UniFi AP Pro
    • Various IP PoE cameras (yet to be purchased)
    • 3 Intel NUC for media PCs
    • Around 10 wireless devices
    • Numerous wired devices around the home
    Will be breaking this down into the following projects
    • Dedicated pfSense box
    • Storage upgrade
    • Storage performance
    • New desk
    • Network reconfigure
    • Server upgrade
     
    #1
  2. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    Dedicated pfSense box

    After looking at various options and scanning eBay for various items I decided it would be easier to just follow Joel's suggestion and buy a prebuilt box. Although I may have gone a little overboard and bought a Dell R230.

    £499 got me the following
    • Xeon E3-1240L v5
    • 8GB DDR4 (single module)
    • 2x 500GB 7200RPM drives
    • iDARC card included
    • Rack rails and front bezel included
    It's in immaculate condition too!

    IMG_8394.jpg

    Proposed spec changes:
    • Add 8GB DDR4 I have lying around (might not be worth it but I can't sell it or use it for anything else)
    • Replace 2x spinning disks with either single SSD or 2x in software RAID1
    • Add Intel X520-DA2
    Only done some quick initial testing power readings are 33w idle in Ubuntu and the noise from this thing is pretty bad, fans seem to be stuck at 50%.
     
    #2
    T_Minus likes this.
  3. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,647
    Likes Received:
    1,036
    That R230 is some serious overkill for a dedicated pfSense box!

    Sent from my VS996 using Tapatalk
     
    #3
    Joel likes this.
  4. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    Agreed!

    Will be running a few service such as squid, snort, VPN (up to 4 clients), pfBlockerNG, some inter VLAN routing and traffic monitoring (undecided on which package will be used for this) but still probably a little overpowered.

    Works out at £415 excluding VAT, admittedly I could have got something a little older/cheaper and saved a £100-150!

    One of the main reasons I went for this was the feedback online of it being 'quiet' compared to older generations especially with a 25W TDP CPU, after all this will be living in my house.
     
    #4
  5. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,267
    Likes Received:
    1,258
    33w with that much 'power' available, great setup!
     
    #5
  6. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    Dedicated pfSense box

    Must admit I like this Dell box, iDRAC 8 is a step up from my usual Supermicro boards. Want to update the BIOS? No problem! Network cable in, press a few keys and away it goes.

    Anyway after updating the BIOS to the latest v2.4.3 and the fans are spinning at their minimum 22% PWM. It's incredibly quiet now :)

    Screen Shot 2018-04-10 at 16.32.07.png

    Swapped out the Dell 7.2k rpm drive for a spare SSD and the idle power consumption is hovering between 20-21w!
     
    #6
  7. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    Dedicated pfSense box

    Intel X520-DA2 installed, this caused the fans to spin back up to 33% minimum.

    IMG_8397.jpg

    Lots of reading later I found a discussion regarding changing the minimum setting via IPMI when unsupported cards have been installed. Had a little play and you can get this all the way down to 10%, at which point it becomes questionable if it's even on.

    Use the following at your own risk to force fans at 22%.

    Code:
    ipmitool \
    -I lanplus \
    -H  10.0.20.212 \
    -U root \
    -P root \
    raw 0x30 0x30 0x02 0xff 0x16
    
    Final power consumption is 25w.
     
    #7
  8. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    161
    Likes Received:
    40
    Another benchmark... :)

    I'm using the "ancient" Supermicro x9spu-f board with an E3-1270v2 69w CPU with 2 sticks of RAM, an SSD, a quad port gigabit card (and two more gigabit from onboard) for my pfsense box.

    idle power consumption - 25w :)

    And it cost less than a third of the cost of this box, and has a much faster CPU (I know I know...IPC and what not). Not knocking this box at all, don't get me wrong, my point is pfSense does not need the absolute latest or lot of hardware. It's pretty thrifty. On my box, I still have to configure some the additional services (hello...pf blocker NG), but I've not seen it even warm up yet, and I have symmetric 1gbps at home for my internet.
     
    #8
  9. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    Nice, my remote backup server runs X9SCM-F with E3-1220v2 and 2 sticks of RAM, could never get that below 40w idle! Maybe the power supply is incredibly inefficient at low powers?

    Either way this was a business expense :rolleyes:
     
    #9
  10. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    161
    Likes Received:
    40
    I'm using HP Platinum power supplies though. This one (and some of the others) are using the 450w one, since that's the smallest common slot PSU HP makes. Those are excellent power supplies, literally silent and very efficient.
     
    #10
  11. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,037
    Likes Received:
    287
    Off topic but how do you connect them to ATX systems ?
    I never thought about reusing HP hot plug supplies for regular servers
     
    #11
  12. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    161
    Likes Received:
    40
    That's...convoluted... :)

    The HP PSU and the PDB i.e. DC-DC converter will not fit a non-HP chassis, unless you're willing to mod the chassis a bit. I was, so I used them.

    The PSU itself is easy to hack. Just two wires to be shorted on the RPS connector and it acts like a regular ATX PSU. The 24 pin/8 pin EPS/4 pin CPU connectors are all standard. Instead of regular molex connectors for other stuff, these PSUs have 2x 10 pin connectors. The wires in them are all standard 5v/12v, so they are easy to mod and convert to molex (or anything else you may need).

    The good news is that the PSUs and PDBb are dead cheap for platinum level. I recently bought 10 more PDBs on fleabay for $60 shipped... :) Bought 10 of the 450w PSUs for about $120.

    In comparison, a single Supermicro PDB can be over $100 and the 500w platinum PSUs still go for about $50.
     
    #12
    nry likes this.
  13. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    Storage Upgrade

    After a few disasters with deliveries from eBuyer
    package 1 - e
    package 2 - e

    package 3 arrived today and the drives finally had adequate packaging! Imgur

    8x Seagate IronWolf 12TB drives
    2x Optane 32GB drives

    IMG_8448.jpg

    Currently doing my usual 'burn in' test on the new drives which is keeping my office nice and warm!
     
    #13
    Evan likes this.
  14. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,671
    Likes Received:
    819
    Nice, what kinda config ya looking at for that stg goodness? I am close to pulling the trigger on some 3.5" stg real-estate, this just makes me wanna go BUY BUY BUY! Good job! :-D
     
    #14
  15. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    Will be running RAID Z2 on FreeNAS, although I'm tempted to give Proxmox a go over ESXi.

    One of the Optane drives will be used to test ZIL performance, not got high expectations for it. Was cheap, figured why not see what it's like.
     
    #15
  16. Joel

    Joel Active Member

    Joined:
    Jan 30, 2015
    Messages:
    689
    Likes Received:
    130
    Just curious, but why Proxmox on top of ESXi? Both are virt platforms so I just do Proxmox on the metal. Proxmox can also do file sharing and ZFS so FreeNAS becomes redundent.
     
    #16
  17. nry

    nry Active Member

    Joined:
    Feb 22, 2013
    Messages:
    307
    Likes Received:
    56
    I meant use it as a replacement for ESXi due to ZFS support
     
    #17
    Joel likes this.
  18. Joel

    Joel Active Member

    Joined:
    Jan 30, 2015
    Messages:
    689
    Likes Received:
    130
    Gotcha. Makes sense now.
     
    #18
  19. Joel

    Joel Active Member

    Joined:
    Jan 30, 2015
    Messages:
    689
    Likes Received:
    130
    I'd definitely recommend Proxmox over FreeNAS if you know what you're doing. Main benefit of FreeNAS is you have the pretty GUI to do your ZFS stuff. BSD-based is a plus too.

    Proxmox isn't as beginner friendly but as long as you're comfortable with CLI you'll be fine. Just make sure you schedule scrubs, SMART tests, etc.
     
    #19
  20. Mishka

    Mishka Member

    Joined:
    Apr 30, 2017
    Messages:
    56
    Likes Received:
    6
    That's a whole lotta storage :D what case is going to be used for your storage server?
     
    #20
Similar Threads: 2018 Upgrading
Forum Title Date
DIY Server and Workstation Builds So.. need help getting into a Quanta Windmill in 2018 :-P Jan 10, 2018
DIY Server and Workstation Builds Workstation 2018: Project Dump Intel - Threadripper Edition Jan 3, 2018
DIY Server and Workstation Builds Buying & Upgrading used workstations - first questions Dec 5, 2017
DIY Server and Workstation Builds Upgrading a 10yo server. Am I doing it right? Mar 16, 2017
DIY Server and Workstation Builds Upgrading 96TB Colocation. Need some ideas for making higher IOPS Feb 2, 2017

Share This Page