Hello everyone! I came across STH website about a year or so ago. I am finally looking at the forum posts. I’m super excited to have found a place that talks about enterprise grade server hardware for the home. I’m here to ask for advice from you experienced guys on what to buy for my needs. I’m not sure if I’m going overboard on the specs or if some things are going to need improvement. I have been working on this project for 5 years now, but kept changing my plans. I might end up making several different threads for the questions I have, but I’m needing advice on
· Building a rack – need help finding affordable rack rails with square holes in them so I can use cage nuts. Planning on making the rack out of wood to keep the $$ down and buy (4) square hole rack rails to mount inside. I’m also custom making the rack so it can fit it in the closet in my office. The closet has a narrow door so I couldn’t get a regular half rack inside.
· ESXi box/storage solution
· Cooling in the closet/room
I’m looking for some suggestions on what to do. I’m really liking ZFS and was recently going to do an ESXi white box and have it connected to a SAN white box with a FC HBA p2p setup. I have already bought some stuff and was going to build a rack out of wood and put drawer slides for the desktop cases I bought. I have since then decided to build a rack with rails so I can use all rackmount equipment. The other reason for this project is to allow me to organize all the data I have collected through the years. It’s spanned across multiple drives and isn’t organized.
Always on VMs
1. 2012 r2 DC – DNS & DHCP
2. 2012 r2 - File/Print server
3. Windows VM – PRTG
4. Win 7 DVR – Avigilon Security Camera DVR
5. Win 8 – PLEX
6. Win 8 for downloads
7. Win 7 – Serva PXE Server or similar
8. vCenter – Not sure if I do appliance or Win VM
9. 2012 r2 – IOT ASP.net Project
10. Mythbuntu
Security VMs – will probably be on most of the time….
1. Alien Vault
2. Linux VM with BRO
3. Security Onion?
4. Linux honeypot?
Sometimes used VMs
1. Xubuntu/CentOS- Steam game server
2. Win 7 – Teamspeak 3/Murmur
3. Win 7/8 - Cisco VIRL – or MABE OVA (they have one right??)
4. Win 8 – GNS3
5. Maybe a few TurnKey Linux appliances
6. Random Linux VM or 2
7. Win 7/8 for learning to code
8. Random Win 7/8 VM or 2
SAN box – Not sure if I want to virtualize this on its own hardware or run as bare metal
· Some flavor of OpenSolaris (Illumos/OpenIndiana) to run napp-it for ZFS, was looking at Nexenta
Network equipment
1. pfSense box – It’s a core 2 duo box – Love this thing!! May look at virtualizing it or run physical and virtual.
2. Cisco 3560G - 24 1-Gig POE ports – probably add another switch or 2 (3750e and a 10GB switch)
3. D-Link Wireless router – used as WAP – Need to change out… looking at Cisco Aironet 3502I AIR-CAP3502I-A-K9
4. ASA 5505 - Used as a VPN Concentrator
5. Cisco 2801 – Not sure what I’m going to do with this… (Maybe get some DSPs and run CME on it) maybe some other fun routing stuff…
6. HDHomeRun OTA Network tuner
UPSs
· Have a few APC UPSs laying around. I have a 1000VA tower unit
· Looking to buy a 120v 1500 VA UPS with LCD and network management card
PC hardware that I own
· (2) Fractal Design Define R4 cases
· 750W corsair PSU
· Storage white box (bought from a friend)
· HP ML301 G5p
· (9) 3TG HGST enterprise 7k SATA Drives (HUS724030ALA640 - 0F14689)
· A few consumer grade SSDs
· (2) E5-2670 CPUs – was originally looking at going UP setup. Started at X34xx series then looked at E5-16xx v1/v2 series. After looking at prices of the E5-1650 v2 and the limited number of ram slots, I decided I could get 2 of the E5-2670s for way less and buy less dense Ram sticks to get the same amount I would need on the UP boards.
· Supermicro board – E-ATX or smaller – Might use my NORCO case for now..? I’m debating how much stuff I want the motherboard to have onboard instead of buying PCI-E cards to add on (Onboard i350-T4, LSI SAS controller…) if I change out my case to a Supermicro one I could go with an EE-ATX board (won’t fit in my NORCO case and will have more mem slots)… I’m trying to make this the best bang for your buck.
· IBM M1015 or equivalent – may need 2 - Will be used in an AIO build.
· Qlogic FC HBA – need if I have a separate storage box
· Intel i350-t4 – might add one anyway if I want to pass some NICs to some VMs and have a dedicated port where I don’t want to use VLANs
· 64GB (8 x 8GB) Registered ECC DDR3 – seems to be the best bang for your buck and would leave we with 8 open ram slots. Should that be enough to run all my VMs and a ZFS VM? I know ZFS is memory hungry… It also depends on how I split up my drives..? (1 RAIDZ2 w/ 9 3TB drives for 21TB total or 2 RAIDZ2 pools.) I also need to determine if I’m going to get enough IOPs with the ZFS setup. I am willing to completely redesign, throw the idea out, or add some SSDs for ZIL and other stuff. As far as the Avigilon Security Camera DVR, I’m planning on using 4 500GB drives in a separate ZFS pool, or passing a raid card through to the VM. I could even move it to a bare metal box too.
ZFS bare metal white box – storage white box listed above
· You think 16 GB of RAM will cut it? Should the i3 be ok?
Questions to answer
1. Make an AIO white box or separate it?
2. Is there enough RAM in the AIO box? I’m fine with things being over provisioned like RAM except for the ZFS VM. This is all for use at home, so I don’t think thing like the DC or FS will need much RAM and will probably set to dynamic with most everything else.
3. Do you see any issues with disk IOPs being too low? I have never used a RAID 6 array let alone a RAIDZ2 array. I originally was going to use NexentaStor CE with (6) 3TB drives. I also have a dedicated AMD APU running OMV with (4) 3TB drives in RAID 5. I decided to get rid of the OMV and move the drives to a bigger ZFS pool… Figured more spindles more IOPs
4. Do you have a similar setup at home? How much power do you think this will consume?
5. Is it going to generate a lot of heat? See question 4.
6. Do you think I will like the performance of the system? I know it’s tough to answer this one.
7. What hardware would you change out?
8. Have anything to sell??
I know it’s a lot going on in this post. I have been writing and organizing my thoughts for the last 4 hours. Hopefully I haven’t rambled on too much and everything is easy to follow.
Thank you for your time and input!!
-Zack
· Building a rack – need help finding affordable rack rails with square holes in them so I can use cage nuts. Planning on making the rack out of wood to keep the $$ down and buy (4) square hole rack rails to mount inside. I’m also custom making the rack so it can fit it in the closet in my office. The closet has a narrow door so I couldn’t get a regular half rack inside.
· ESXi box/storage solution
· Cooling in the closet/room
I’m looking for some suggestions on what to do. I’m really liking ZFS and was recently going to do an ESXi white box and have it connected to a SAN white box with a FC HBA p2p setup. I have already bought some stuff and was going to build a rack out of wood and put drawer slides for the desktop cases I bought. I have since then decided to build a rack with rails so I can use all rackmount equipment. The other reason for this project is to allow me to organize all the data I have collected through the years. It’s spanned across multiple drives and isn’t organized.
ESXi box VMs
Always on VMs
1. 2012 r2 DC – DNS & DHCP
2. 2012 r2 - File/Print server
3. Windows VM – PRTG
4. Win 7 DVR – Avigilon Security Camera DVR
5. Win 8 – PLEX
6. Win 8 for downloads
7. Win 7 – Serva PXE Server or similar
8. vCenter – Not sure if I do appliance or Win VM
9. 2012 r2 – IOT ASP.net Project
10. Mythbuntu
Security VMs – will probably be on most of the time….
1. Alien Vault
2. Linux VM with BRO
3. Security Onion?
4. Linux honeypot?
Sometimes used VMs
1. Xubuntu/CentOS- Steam game server
2. Win 7 – Teamspeak 3/Murmur
3. Win 7/8 - Cisco VIRL – or MABE OVA (they have one right??)
4. Win 8 – GNS3
5. Maybe a few TurnKey Linux appliances
6. Random Linux VM or 2
7. Win 7/8 for learning to code
8. Random Win 7/8 VM or 2
SAN box – Not sure if I want to virtualize this on its own hardware or run as bare metal
· Some flavor of OpenSolaris (Illumos/OpenIndiana) to run napp-it for ZFS, was looking at Nexenta
Hardware
Network equipment
1. pfSense box – It’s a core 2 duo box – Love this thing!! May look at virtualizing it or run physical and virtual.
2. Cisco 3560G - 24 1-Gig POE ports – probably add another switch or 2 (3750e and a 10GB switch)
3. D-Link Wireless router – used as WAP – Need to change out… looking at Cisco Aironet 3502I AIR-CAP3502I-A-K9
4. ASA 5505 - Used as a VPN Concentrator
5. Cisco 2801 – Not sure what I’m going to do with this… (Maybe get some DSPs and run CME on it) maybe some other fun routing stuff…
6. HDHomeRun OTA Network tuner
UPSs
· Have a few APC UPSs laying around. I have a 1000VA tower unit
· Looking to buy a 120v 1500 VA UPS with LCD and network management card
PC hardware that I own
· (2) Fractal Design Define R4 cases
· 750W corsair PSU
· Storage white box (bought from a friend)
o Norco 4U 24 HDD bay case – Similar to this
o i3-2120T
o Supermicro motherboard
o 16gb ECC unbuffered DDR3
o Raid card – not going to use
o i3-2120T
o Supermicro motherboard
o 16gb ECC unbuffered DDR3
o Raid card – not going to use
· HP ML301 G5p
o Xeon E3120
o 8GB unbuffered ECC DDR2
o 3X 300GB 15k SAS in Raid 0
o 8GB unbuffered ECC DDR2
o 3X 300GB 15k SAS in Raid 0
· (9) 3TG HGST enterprise 7k SATA Drives (HUS724030ALA640 - 0F14689)
· A few consumer grade SSDs
ESXi white box hardware to buy
· (2) E5-2670 CPUs – was originally looking at going UP setup. Started at X34xx series then looked at E5-16xx v1/v2 series. After looking at prices of the E5-1650 v2 and the limited number of ram slots, I decided I could get 2 of the E5-2670s for way less and buy less dense Ram sticks to get the same amount I would need on the UP boards.
· Supermicro board – E-ATX or smaller – Might use my NORCO case for now..? I’m debating how much stuff I want the motherboard to have onboard instead of buying PCI-E cards to add on (Onboard i350-T4, LSI SAS controller…) if I change out my case to a Supermicro one I could go with an EE-ATX board (won’t fit in my NORCO case and will have more mem slots)… I’m trying to make this the best bang for your buck.
o X9DR7-LN4F or maybe one with JBOD to if I do an AIO box. Found on eBay for $415. There are similar models for less, but they don’t have and PCI-E x16 slots… May want later for GFX stuff.
· IBM M1015 or equivalent – may need 2 - Will be used in an AIO build.
· Qlogic FC HBA – need if I have a separate storage box
· Intel i350-t4 – might add one anyway if I want to pass some NICs to some VMs and have a dedicated port where I don’t want to use VLANs
· 64GB (8 x 8GB) Registered ECC DDR3 – seems to be the best bang for your buck and would leave we with 8 open ram slots. Should that be enough to run all my VMs and a ZFS VM? I know ZFS is memory hungry… It also depends on how I split up my drives..? (1 RAIDZ2 w/ 9 3TB drives for 21TB total or 2 RAIDZ2 pools.) I also need to determine if I’m going to get enough IOPs with the ZFS setup. I am willing to completely redesign, throw the idea out, or add some SSDs for ZIL and other stuff. As far as the Avigilon Security Camera DVR, I’m planning on using 4 500GB drives in a separate ZFS pool, or passing a raid card through to the VM. I could even move it to a bare metal box too.
ZFS bare metal white box – storage white box listed above
· You think 16 GB of RAM will cut it? Should the i3 be ok?
Questions to answer
1. Make an AIO white box or separate it?
a. If separating it out what should be separate? The ZFS box or the DVR or both?
2. Is there enough RAM in the AIO box? I’m fine with things being over provisioned like RAM except for the ZFS VM. This is all for use at home, so I don’t think thing like the DC or FS will need much RAM and will probably set to dynamic with most everything else.
3. Do you see any issues with disk IOPs being too low? I have never used a RAID 6 array let alone a RAIDZ2 array. I originally was going to use NexentaStor CE with (6) 3TB drives. I also have a dedicated AMD APU running OMV with (4) 3TB drives in RAID 5. I decided to get rid of the OMV and move the drives to a bigger ZFS pool… Figured more spindles more IOPs
4. Do you have a similar setup at home? How much power do you think this will consume?
5. Is it going to generate a lot of heat? See question 4.
6. Do you think I will like the performance of the system? I know it’s tough to answer this one.
7. What hardware would you change out?
8. Have anything to sell??
I know it’s a lot going on in this post. I have been writing and organizing my thoughts for the last 4 hours. Hopefully I haven’t rambled on too much and everything is easy to follow.
Thank you for your time and input!!
-Zack
