2018 - Upgrading the toys

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nry

Active Member
Feb 22, 2013
312
61
28
Got the itch to upgrade various things at the moment, all while renovating a whole house. Turning into a hectic year!

I have come a long way in recent years, see my original build log at https://forums.servethehome.com/ind...ld-vsphere-cluster-and-network-overhaul.1712/ where I probably bought items without too much thought into how they will be used. Will try not continue this but given I probably have a spending problem, I make no promises.

Recently posted a rough outline for upgrade options here https://forums.servethehome.com/index.php?threads/homelab-update-options-advise.18954/

This thread probably won't be on the same level as some of the incredible setups posted here, but still hope some find it at least remotely interesting :rolleyes:

Current setup
Rack

  • APC UPS - SMT1000RMI2U
  • APC PDU ZeroU
  • Dell PowerConnect 5525P
  • Dell PowerConnect 5524 (stacked with above)
  • MikroTik CRS317-1G-16S+RM - not used currently, this will be the core switch
  • Main Server running ESXi 6 - runs 24/7
    • Supermicro X9SRL-F
    • Xeon E5-1620v2
    • 128GB ECC DDR3
    • 2x M1015 HBA
    • Samsung Evo 840 120GB boot drive
    • Samsung SM961 1TB NVMe - primary datastore
    • Samsung Evo 850 1TB - local backup datastore
    • WD 6TB Red - Media storage due to storage constraints on ZFS pool
    • 12x Hitachi/HGST 3TB 5k3000 drives (might be 1 or 2 7K3000 drives mixed in)
    • Intel X520-DA2 SFP+ NIC
  • Secondary server running ESXi 6 or what ever OS I need to experiment with
    • Asus Z9PE-D8 WS (with IPMI chip)
    • 2x Xeon E5-2660
    • 128GB ECC DDR3
    • Mixed old SSDs for testing
    • 3-4 Hitachi 3TB 7K3000 drives
    • Asus HD5450 (used to passthrough to VM)
    • Intel X520-DA2 SFP+ NIC
Desk/Workstation
  • Desktop running Mac OS
    • Gigabyte GA-Z97X-UD3H
    • i7 4790K
    • 32GB Corsair DDR3
    • Samsung SM951 512GB
    • Samsung 940 Pro 256 GB (Windows boot drive)
    • Corsair CX650
    • GTX 980
    • Myricom 10G-PCIE2-8B2-2S (dual port SFP+ card)
    • Generic bluetooth/WiFi card for Mac
  • 3x Dell P2715Q Monitors
  • Apple Keyboard/Trackpad
  • Razor Mamba Mouse
  • Audiolab 8000P Amp
  • Audolab M-DAC
  • B&W 685 S2 speakers
  • 15" MacBook Pro 2014 (i7, 16GB RAM, 512GB SSD)
    • Sonnect Echo Express with Myricom 10G-PCIE-8B-S (1 port SFP+ card)
Misc
  • UniFi AP Pro
  • Various IP PoE cameras (yet to be purchased)
  • 3 Intel NUC for media PCs
  • Around 10 wireless devices
  • Numerous wired devices around the home
Will be breaking this down into the following projects
  • Dedicated pfSense box
  • Storage upgrade
  • Storage performance
  • New desk
  • Network reconfigure
  • Server upgrade
 

nry

Active Member
Feb 22, 2013
312
61
28
Dedicated pfSense box

After looking at various options and scanning eBay for various items I decided it would be easier to just follow Joel's suggestion and buy a prebuilt box. Although I may have gone a little overboard and bought a Dell R230.

£499 got me the following
  • Xeon E3-1240L v5
  • 8GB DDR4 (single module)
  • 2x 500GB 7200RPM drives
  • iDARC card included
  • Rack rails and front bezel included
It's in immaculate condition too!

IMG_8394.jpg

Proposed spec changes:
  • Add 8GB DDR4 I have lying around (might not be worth it but I can't sell it or use it for anything else)
  • Replace 2x spinning disks with either single SSD or 2x in software RAID1
  • Add Intel X520-DA2
Only done some quick initial testing power readings are 33w idle in Ubuntu and the noise from this thing is pretty bad, fans seem to be stuck at 50%.
 
  • Like
Reactions: T_Minus

nry

Active Member
Feb 22, 2013
312
61
28
That R230 is some serious overkill for a dedicated pfSense box!

Sent from my VS996 using Tapatalk
Agreed!

Will be running a few service such as squid, snort, VPN (up to 4 clients), pfBlockerNG, some inter VLAN routing and traffic monitoring (undecided on which package will be used for this) but still probably a little overpowered.

Works out at £415 excluding VAT, admittedly I could have got something a little older/cheaper and saved a £100-150!

One of the main reasons I went for this was the feedback online of it being 'quiet' compared to older generations especially with a 25W TDP CPU, after all this will be living in my house.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
33w with that much 'power' available, great setup!
 

nry

Active Member
Feb 22, 2013
312
61
28
Dedicated pfSense box

Must admit I like this Dell box, iDRAC 8 is a step up from my usual Supermicro boards. Want to update the BIOS? No problem! Network cable in, press a few keys and away it goes.

Anyway after updating the BIOS to the latest v2.4.3 and the fans are spinning at their minimum 22% PWM. It's incredibly quiet now :)

Screen Shot 2018-04-10 at 16.32.07.png

Swapped out the Dell 7.2k rpm drive for a spare SSD and the idle power consumption is hovering between 20-21w!
 

nry

Active Member
Feb 22, 2013
312
61
28
Dedicated pfSense box

Intel X520-DA2 installed, this caused the fans to spin back up to 33% minimum.

IMG_8397.jpg

Lots of reading later I found a discussion regarding changing the minimum setting via IPMI when unsupported cards have been installed. Had a little play and you can get this all the way down to 10%, at which point it becomes questionable if it's even on.

Use the following at your own risk to force fans at 22%.

Code:
ipmitool \
-I lanplus \
-H  10.0.20.212 \
-U root \
-P root \
raw 0x30 0x30 0x02 0xff 0x16
Final power consumption is 25w.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Another benchmark... :)

I'm using the "ancient" Supermicro x9spu-f board with an E3-1270v2 69w CPU with 2 sticks of RAM, an SSD, a quad port gigabit card (and two more gigabit from onboard) for my pfsense box.

idle power consumption - 25w :)

And it cost less than a third of the cost of this box, and has a much faster CPU (I know I know...IPC and what not). Not knocking this box at all, don't get me wrong, my point is pfSense does not need the absolute latest or lot of hardware. It's pretty thrifty. On my box, I still have to configure some the additional services (hello...pf blocker NG), but I've not seen it even warm up yet, and I have symmetric 1gbps at home for my internet.
 

nry

Active Member
Feb 22, 2013
312
61
28
Nice, my remote backup server runs X9SCM-F with E3-1220v2 and 2 sticks of RAM, could never get that below 40w idle! Maybe the power supply is incredibly inefficient at low powers?

Either way this was a business expense :rolleyes:
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Nice, my remote backup server runs X9SCM-F with E3-1220v2 and 2 sticks of RAM, could never get that below 40w idle! Maybe the power supply is incredibly inefficient at low powers?

Either way this was a business expense :rolleyes:
I'm using HP Platinum power supplies though. This one (and some of the others) are using the 450w one, since that's the smallest common slot PSU HP makes. Those are excellent power supplies, literally silent and very efficient.
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
I'm using HP Platinum power supplies though. This one (and some of the others) are using the 450w one, since that's the smallest common slot PSU HP makes. Those are excellent power supplies, literally silent and very efficient.
Off topic but how do you connect them to ATX systems ?
I never thought about reusing HP hot plug supplies for regular servers
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Off topic but how do you connect them to ATX systems ?
I never thought about reusing HP hot plug supplies for regular servers
That's...convoluted... :)

The HP PSU and the PDB i.e. DC-DC converter will not fit a non-HP chassis, unless you're willing to mod the chassis a bit. I was, so I used them.

The PSU itself is easy to hack. Just two wires to be shorted on the RPS connector and it acts like a regular ATX PSU. The 24 pin/8 pin EPS/4 pin CPU connectors are all standard. Instead of regular molex connectors for other stuff, these PSUs have 2x 10 pin connectors. The wires in them are all standard 5v/12v, so they are easy to mod and convert to molex (or anything else you may need).

The good news is that the PSUs and PDBb are dead cheap for platinum level. I recently bought 10 more PDBs on fleabay for $60 shipped... :) Bought 10 of the 450w PSUs for about $120.

In comparison, a single Supermicro PDB can be over $100 and the 500w platinum PSUs still go for about $50.
 
  • Like
Reactions: nry

nry

Active Member
Feb 22, 2013
312
61
28
Storage Upgrade

After a few disasters with deliveries from eBuyer
package 1 - e
package 2 - e

package 3 arrived today and the drives finally had adequate packaging! Imgur

8x Seagate IronWolf 12TB drives
2x Optane 32GB drives

IMG_8448.jpg

Currently doing my usual 'burn in' test on the new drives which is keeping my office nice and warm!
 
  • Like
Reactions: Evan

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Nice, what kinda config ya looking at for that stg goodness? I am close to pulling the trigger on some 3.5" stg real-estate, this just makes me wanna go BUY BUY BUY! Good job! :-D
 

nry

Active Member
Feb 22, 2013
312
61
28
Will be running RAID Z2 on FreeNAS, although I'm tempted to give Proxmox a go over ESXi.

One of the Optane drives will be used to test ZIL performance, not got high expectations for it. Was cheap, figured why not see what it's like.
 

Joel

Active Member
Jan 30, 2015
850
191
43
42
Will be running RAID Z2 on FreeNAS, although I'm tempted to give Proxmox a go over ESXi.

One of the Optane drives will be used to test ZIL performance, not got high expectations for it. Was cheap, figured why not see what it's like.
Just curious, but why Proxmox on top of ESXi? Both are virt platforms so I just do Proxmox on the metal. Proxmox can also do file sharing and ZFS so FreeNAS becomes redundent.
 

Joel

Active Member
Jan 30, 2015
850
191
43
42
I'd definitely recommend Proxmox over FreeNAS if you know what you're doing. Main benefit of FreeNAS is you have the pretty GUI to do your ZFS stuff. BSD-based is a plus too.

Proxmox isn't as beginner friendly but as long as you're comfortable with CLI you'll be fine. Just make sure you schedule scrubs, SMART tests, etc.
 

Mishka

Active Member
Apr 30, 2017
101
34
28
London, UK
Storage Upgrade

After a few disasters with deliveries from eBuyer
package 1 - e
package 2 - e

package 3 arrived today and the drives finally had adequate packaging! Imgur

8x Seagate IronWolf 12TB drives
2x Optane 32GB drives

View attachment 8336

Currently doing my usual 'burn in' test on the new drives which is keeping my office nice and warm!
That's a whole lotta storage :D what case is going to be used for your storage server?