Home Setup - Design changes

[marcoi]

Well I'll be closing on a home in July and with that I want to consider redoing my home setup. I will break the home setup into categories and also current setup and future setup. I would like input on future setup, thus the discussion part of this thread.

Categories
1. Network
2. VM Servers for home
3. Storage
4. Security
5. IP Cameras
6. Other Misc

If you want to add to the discussion, please specify the category to keep the thread understandable.
I also plan to use the thread as work in progress as I start re-doing the network in the new home.


Old Home:
State: NH
Weather: Cold 9 months of year / warm in summer
Electricity: Average 18 cents per KWH
Internet: Comcast Cable - 300 mbs service


New Home:
State: FL
Weather: Hot 9 months of year, warm in winter
Electricity: Average 14 cents per KWH (when including tax and surcharges)
Internet: ATT Giga Fiber

 

[marcoi]

1. Networking
=========================================================================

Current:
Network interface:

• 1 GB for all wired computers.
  
• 10 GB for storage to servers.
  
• AC1750 Wifi Access points.

Hardware:

• 3 x Dell PowerConnect 5524 24-Port Ethernet Gigabit Switches. I use the 10 GB SFP+ ports to run the 10 GB between servers and use HDMI cable to connect the switches.
• 2 X TP-Link AC1750 Wireless Wi-Fi Access Point for Wifi.
  
• Servers are using RT8N1 DELL MELLANOX CONNECTX-2 PCIe 10GBe Nics

IP Range:

• 192.x for home and 172.x for development

Future:
Network interface:

• I want to run cat 6? cable in the house and have 2-4 drops in each room. That will ensure i can upgrade from 1 GB to 10 GB easily in the future. The house may already have some drops in place, but i dont know the spec of the wire.

Hardware:

• I would like to get something a little more energy efficient then the dells, maybe with more ports and with 10 GB SFP+ so i can connect the servers as well.

IP Range:

• I dont mind keeping the 192.x setup for home. It will provide me enough IP ranges. What I want to work on is what gives out the DHCP IP address and the communication between the home 192 and dev lab 172 ip address. Any recommendations?

 

[marcoi]

2. VM Servers for home
=========================================================================

Current:
VM Hosts:

• ESXI 6.5 with 6 CPU limit through VMUG

Servers:

• Intel 4u server: 2x E5-2670 CPU / 128 GB Ram / Raid Cards/ 10 GB / USB
  
• Dell R720 2u Server: 2x CPU E5-2648L v2 CPU / 192 GB Ram / 2 LSI 12GB Raid Cards / 10 GB Nic / USB
• Supermicro CSE-836BE1C-R1K03JBOD 3u 16 x 3.5 HDD.
• Intel 2u RMS25CB080 server: 2 x CPU? / 128 GB Ram/ 2x 1050 TI video cards
• Supermicro SuperServer 6027TR-HTFRF 4 Node 2u Server: 8 x E5-2680V2 / 128 GB Total Ram

Setup:

• Dell r720 and intel 4u servers are for my home network. They are running ESXI 6.5 and run all my home VMs. The Dell also has storage VM running.
• Intel 2u and Supermicro 4 node were for mining, but will probably be re-purposed.

Future:
VM Hosts:

• I plan to stay with ESXI and Vcenter for the home server usage.
  
• I might try another VM host for dev lab, to play with.

Servers:

• For now I plan to keep the servers, but i might want to either get smaller servers that are more efficient or upgrade to a newer gen and consolidate. I got plenty of power and ram to run modern VMs and workloads, but the energy usage might make me want to change it up. Plus I like to get new toys.

Setup:

• I'm deciding where to setup the servers. Currently the plan is to use a bedroom so they are in AC controlled area. I would eventually like to build out a data room in the garage and use a mini-split AC for that area to free up the bedroom. but its not high on the priority list.
• I will get a used Rack since i have the room now to store all the servers in.

 

[marcoi]

3. Storage
=========================================================================
Current:
Overview:
I use an AIO setup with my VM servers to handle storage. I use FreeNas to handle raw storage and pass that out to other servers using ISCSI or NFS. I have 4 pools setup. Two of the pools (Data and backup) are used by Windows 2012 R2 Essentials VM which shares out storage and performs backup of my windows servers using the connector software. The other pool is backup site for ESXI VMs and the last is SSD pool for playing with VMs.

Storage:

• I am currently running FreeNas 11.1.x under VM on the Dell r720.
  
• FreeNas VM has 64GB of Ram assigned.
  
• It has two LSI 12 GB cards in IT mode in pass-through mode to the VM.
  
• Internal LSI card handles 6 x 2.5 12 GB cage which was installed into the 720 using 730 parts. I have 6 x : HGST 100GB SAS SSD 2.5'' HDD HUSSL4010BSS600 drives attached.
• External LSI card connects to the Supermicro 3u JBOD which has 8 x 8TB RED drives (from easystore bb)
  
• I have two z1 pools of 3 8TB drives each. One pool is used for data storage, Other for backups.
  
• Two z1 pools are ISCSI out to ESXI host and attached to Windows 2012 R2 Essentials VM. One is for Data 8 TB and the other is for backups also 8TB.
  
• I currently use an intel 900P 280 GB AIC in the Dell r720 and hosted in ESXI as a datastore. I pass 4 30 GB drives from the intel 900p into FreeNas VM to use as slog drives.

Future:
Overview:
I dont know if I want to stay with FreeNas or move over to QNAP box. I like the idea of QNAP, but also like the freedom of FreeNas. I dont know how the two would compare with in terms of speed of storage pools. With FreeNas, I can add more memory and with intel 900p acting as slog, its very fast. I also have 10 GB internal ESXI network and also 10 GB out of ESXI to the switches etc setup.

If I stay with FreeNas, I think I want to change the pools over to z2 setup and possible upgrade to two separate intel 900p drives?

I also currently using Windows 2012 R2 Essentials for handling files on the network. Most of my computers are windows OS based. I run a few Linux VMs but access storage using Freenas build for Dev lab separate from home freenas build. I also use Windows server built in backup for automating backup of all my computers.

I been thinking of updating to windows 2016 R2 essentials. but that has a licensing cost. What do others use for network storage and windows OS backup. I like things that are simple set and forget.

 

[marcoi]

4. Security
=========================================================================
Current:
Overview: I currently run Sophos UTM 9.x as VM to handle my internet security. I been running it for years and have it configured with application blocks, country blocks, scans, filters, etc.

For Antivirus: I use Avast Business Pro Plus. I also been using various Avast antivirus for years. I have it installed on all my physical computers, a few VMs and my windows 2012 r2 essentials server.

Future:
Overview: I would like to move to the new UTM from Sophos for home usage. There is no more IP limits with the new version. The PITA is there is no migration option from old UTM to the new so everything needs to be redone.

I would like to also learn how to properly setup vlans to keep the various traffic in my network separate. Any guides or help with that would be useful.

Also not sure how well the UTM will do with Fiber Giga internet connection. I'm assuming if i up the resources IE add more ram/cpu cores it will be fine.

Finally for Wifi- any suggestions on best practices to avoid them getting hacked into?

 

[marcoi]

5. IP Cameras
=========================================================================

I got nothing for IP Cameras. I definitely want to run several around the house that are POE capable and 1080 resolution with night vision. I dont want to use camera that have cloud service. I want the cameras to store locally using either free camera software or QNAP addon- if i go with that as storage option.

Any suggestions would be great both for software, hardware and general tips, issues.

6.13.18 - I'm thinking of using ZoneMinder for IP Camera software. Anyone using it now?

 

[marcoi]

6. Other Misc
=========================================================================

IP Phones: I'm currently running Wazo PBX as a VM with google voice and zoiper app on my cell phone for business line. I need to upgrade my base Wazo install to new version, I also want to get and IP based physical phone at some point and only use the app when im a way.

 

[mjygvfesz]

I wired my home for networking shortly after purchasing it as well. so for the network piece I would recommend a POE switch especially if you plan to do IP cameras. it may increase your initial investment but in the end it will be the mass of POE converters you have either at your patch or between the wall ports and the device. I personally went with Ubiquity for my network infrastructure, I did a 48 port 500W POE and one of there XG 10G switches.

 

[marcoi]

I wired my home for networking shortly after purchasing it as well. so for the network piece I would recommend a POE switch especially if you plan to do IP cameras. it may increase your initial investment but in the end it will be the mass of POE converters you have either at your patch or between the wall ports and the device. I personally went with Ubiquity for my network infrastructure, I did a 48 port 500W POE and one of there XG 10G switches.

How much power do the switches use?
I will probably get a 24 port POE switch. I dont see ever needing more then 24. I'm thinking 12 for Camera, probably really only run 8. Then 2-4 for Wifi access points. One for IP phone.

I got to figure out a good location for patch panel. I dont know if I should plan it in a bedroom closet or some other closet. Might be best to have the patch panel in the roof space, then have two runs one into bedroom, the other into garage, then i can switch whenever needed.

 

[mjygvfesz]

Power usage based on POE usage but the minimum with no POE devices is 64W for the 48 port and 56 for the 10G switch. I personally went with 4 ports per bedroom (3 in my case) and 4 for the living room area. I figured computer, cable box/smart tv, and a console or 2 per area was going to work out well. that gave 16 before I included my one AP, add in all the other lines for IPMI and basic uplink drove me to the 48 port version. the other driving force was that the 48 port version has 2 x 10G SFP+ for links to the 10G server switch.

The location I went with was in my garage as it provided a nice insulated wall between it and the rest of the house to help drown out the noise. the proximity to the breaker and all those things makes it easy if you need to add a circuit for 30 amp or 240V service. Concrete floor help if you go to a standing rack, exposed studs help for wall mounting.

 

[marcoi]

4. Security
=========================================================================
Current:
Overview: I currently run Sophos UTM 9.x as VM to handle my internet security. I been running it for years and have it configured with application blocks, country blocks, scans, filters, etc.

For Antivirus: I use Avast Business Pro Plus. I also been using various Avast antivirus for years. I have it installed on all my physical computers, a few VMs and my windows 2012 r2 essentials server.

Future:
Overview: I would like to move to the new UTM from Sophos for home usage. There is no more IP limits with the new version. The PITA is there is no migration option from old UTM to the new so everything needs to be redone.

I would like to also learn how to properly setup vlans to keep the various traffic in my network separate. Any guides or help with that would be useful.

Also not sure how well the UTM will do with Fiber Giga internet connection. I'm assuming if i up the resources IE add more ram/cpu cores it will be fine.

Finally for Wifi- any suggestions on best practices to avoid them getting hacked into?

Well I finally got my servers setup in the new home and found out that Sophos UTM seems to have trouble with the Giga fiber speeds.

Seems to be due to IPS (Intrusion Prevention). I set snoop(sp?) to use 4 cores since I have 4 cores assigned to the VM. that helped a little bit. I also moved the VM to server with high freq CPU and that helped as well.

Right now with 4 Cores, 4 GB Ram and on the faster CPU (2.7GHZ) i am getting the below with IPS on.


With it off:


With direct to modem i seen upto 900 both up and down.

Anyone have suggestions on getting Sophos UTM to run at full speed?

 

[turgin]

Sophos IPS uses snort which is single threaded. So, adding more cores to the guest won't help much as plain old raw GHz which is what you're seeing. I can't say for sure exactly what is needed but I do know that the typical recommendation, on the Sophos forums at least, is 3 GHz minimum. I only have a measly 50/5 cable modem service but I can tell you that Sophos performed noticeably better for me on an X5670 (2.9 GHz) than on an E5-2650 (2 GHz) which correlates with what you're seeing and what I've read. You're going to need "moar GHz" or turn off features to maximize that 1G internet if using UTM.

I don't know if XG performs better in that regard or not, but pfsense certainly seems to. I still prefer UTM myself.

 

[marcoi]

Sophos IPS uses snort which is single threaded. So, adding more cores to the guest won't help much as plain old raw GHz which is what you're seeing. I can't say for sure exactly what is needed but I do know that the typical recommendation, on the Sophos forums at least, is 3 GHz minimum. I only have a measly 50/5 cable modem service but I can tell you that Sophos performed noticeably better for me on an X5670 (2.9 GHz) than on an E5-2650 (2 GHz) which correlates with what you're seeing and what I've read. You're going to need "moar GHz" or turn off features to maximize that 1G internet if using UTM.

I don't know if XG performs better in that regard or not, but pfsense certainly seems to. I still prefer UTM myself.

I had found the same info as you. I might spin up and e3 box i have that core cpu is 3.4ghz and see if sophos runs better on that. I also have XG VM i started to setup which i might be able to test to see if it s better.

 

[marcoi]

Well I finally got my servers setup in the new home and found out that Sophos UTM seems to have trouble with the Giga fiber speeds.

Seems to be due to IPS (Intrusion Prevention). I set snoop(sp?) to use 4 cores since I have 4 cores assigned to the VM. that helped a little bit. I also moved the VM to server with high freq CPU and that helped as well.

Right now with 4 Cores, 4 GB Ram and on the faster CPU (2.7GHZ) i am getting the below with IPS on.
View attachment 8760

With it off:
View attachment 8761

With direct to modem i seen upto 900 both up and down.

Anyone have suggestions on getting Sophos UTM to run at full speed?

Just an update to this. I still am running UTM but upgraded the CPU in my servers to dual e5-2680 v2 which had a higher freq then the older CPU. So far i getting 450ish down and 1G up according to Fast.com. Still testing out the newer version of UTM but havent had time to set it up. ( i have a lot of rules in current deployment and there is not upgrade path).

 

[marcoi]

Finally got around to getting an open rack and set it up last night. Here are some pics.


I still have to do some clean up and may move things around after a while. Right now moving them servers from the old Rack (sound deading) the whole room dropped 6 degs in temp. Prior it was running 92 F, now its at 86 F.

 

[gigatexal]

Just an update to this. I still am running UTM but upgraded the CPU in my servers to dual e5-2680 v2 which had a higher freq then the older CPU. So far i getting 450ish down and 1G up according to Fast.com. Still testing out the newer version of UTM but havent had time to set it up. ( i have a lot of rules in current deployment and there is not upgrade path).

View attachment 9168

what about a i5 2500k 4 cores 4 threads but OC'd to say 4.5 or more ghz? basically a really fast desktop box just for this one thing. It could be quiet with a big noctua cooler too

anyone know if the IDS stuff on pfsense multi-threaded?

 

[marcoi]

what about a i5 2500k 4 cores 4 threads but OC'd to say 4.5 or more ghz? basically a really fast desktop box just for this one thing. It could be quiet with a big noctua cooler too

anyone know if the IDS stuff on pfsense multi-threaded?

I havent decided if I want to branch out to different hardware right now. I think i want to test a bit more with the newer version of Sophos UTM to see how it performs.

Moving Sophos to a standalone box, means another piece of equipment to maintain and run and pay electricity for.

Now that i got my rack in place, I'll be going through my stuff and re-eval it to sell, etc..

Biggest problem is I need to sleep at some point so i can spend endless time tinkering with all this lol.

 

[gigatexal]

I know how that goes. I was once dating a girl long distance and she would call me very, very angry because I had blown a whole saturday tinkering on some project and not once did I think to text or call her. When in the zone I didn't think of sleep or food just going full MacGuyver on things. Kudos on the new place btw

 

[marcoi]

I upgrade my storage to now include 8 x 800GB Micron S630DC SAS SSD. (from great deals thread)
The plan is to setup some pools in my freenas vm and them move all the VMs onto those pools vs keeping them on local ssds.
This will help with vmotion, and also moving in and out new/different hardware till i get a setup im content with.

Doing some testing and marking it here. I'll post more as I go.

Setup:HBA 9300-8i to dell r730 sas3 cage.
Drives updated to latest firmware.

Test1 - Win 7 test VM, HBA pass-through. 1 Drive CDM defaults - but time set to 0 sec wait.


Test1 - Win 7 test VM, HBA pass-through. 4 Drives - windows stripe raid CDM defaults - but time set to 0 sec wait.

 

[marcoi]

FreeNas testing
ISCSI of zvol to ESXI host.
100 GB drive thin provisioned passed to win7 VM.
FreeNas sync for pool set to standard.



Q32 threads 1



q32 threads 2



q32 threads 16