The PowerRouter: One beast to rule them all

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

marcoi

Well-Known Member
Apr 6, 2013
1,532
288
83
Gotha Florida
I want to build a new server to run multiple instances of Router/Firewall VMs. The purpose is to have all the networking and a few other VMs running on the box so when i need to bring down a server for maintenance i dont lose networking.

VMs so Far
  1. ATT PFsense - I replaced my gigafiber router with pfsense VM recently. This VM connects my network to ATT service and provides WAN in. It also provides subnet for IOT wireless AP.
  2. Sophos UTM9 - I use this for home network to reduce/control/block internet garbage. I have a lot of rules setup so I dont plan on replacing it with something else anytime soon.
  3. PFSense_External1-4: I want to setup between 1-4 pfsense vms for each of my external IP adddress. I have 5 available. Each PFsense will deal with it's own DHCP and firewall rules for services running on that IP.
  4. Backup WS2012R2: Provide home network DNS
  5. WS2012R2 Essentials: Provides main DNS and backup/storage access. Storage is iSCSI attached back to freenas vm.
  6. W10 tester: Just to access various Pfsense images.

Build Out Desires.
CPU: 4+Core/ 2.8Gh+ or Higher Freq(prefer 3-4 Ghz base)
RAM: 64 Min - 128 Max DDR4
Nics: 8-12 ports either onboard or via intel nics.
Features:
  • IPMI supported.
  • CPU acceleration that helps pfsense, etc.
  • Runs ESXI
  • Allows Pass-through


Build out Options.
1. Recycle Server - Intel S2600CP w/ Intel P4000 Case
I have my retired intel server. Currently i have a cheap e5-26xxx v2 cpu and a few sticks of RAM. The Motherboard only has 2 ports. With one CPU installed I lose access to 1 PCIE slot. I also have an older intel 4port nic installed. I can also take another intel 4port nic from current server running PFsense images and move it into this server.
Pros: - Cheap option, might need to hunt down faster base CPU and possibly another intel 4 port nic.
Cons: Idles easily around 130-150watts as configured.

2. SuperServer 1019C-FHTN8 Server Build. The cost of building this out with 4/6 core e-2100 CPU and 128GB Ram will be around 2k give or take. I like that the MB has 8 intel nic ports built in. I would use the extra pcie slot for another 4port nic or 1 sfp+ 10GB. Storage would be a local 3-4TB SSD setup.
Concerns:
  • Long term reuse ability. This is very specific for network applications. If i change my setup down the line, this would be limited use item.
  • Costs
  • I dont know power idle consumption. I assume less then option 1? but by how much?
  • I dont know if i can pass through the onboard nics.

3. Setup a new 3rd Gen AMD Ryzen based server using Asus Pro WS X570-ACE motherboard. I would need to use 3 pcie slots for Nics. I would also get ECC Ram. I havent priced this out but figuring about 2k as well. maybe less depending on memory/cpu and if i reuse the two intel 4port nics i have.
Concerns:
  • Power usage for idle
  • ESXI issues
  • Newer tech - havent played with
  • MB IPMI maybe more limited then server board?

So Im still in research mode and figured i put this out there for community to discuss. Let me know your thoughts.
 
Last edited:

j_h_o

Active Member
Apr 21, 2015
644
179
43
California, US
What kind of switch(es) do you have? Do you really need so many NIC ports? Can you just use VLANs and fewer ports?
I've moved to a single 10Gbps port that carries all the VLANs I need, for my VMs.
 

marcoi

Well-Known Member
Apr 6, 2013
1,532
288
83
Gotha Florida
What kind of switch(es) do you have? Do you really need so many NIC ports? Can you just use VLANs and fewer ports?
I've moved to a single 10Gbps port that carries all the VLANs I need, for my VMs.
I have two Dell PowerConnect 5524 switchs connected to each other using hdmi cable. Each switch has 10 GB SFP+ connector. I have two used already.

I currently use the 10GB for storage connections and vmotion.

for my ATT PFsense VM, it needs passthrough nic atleast for connection to gigafiber line to authenticate.
I'm not sure if i can do VNics and VLANS for some of the other connections between other pfsense vms and rest of the network..

I would have to test more with how virtual nics work within pfsense vm vs pass-thought.
Plus i dont know if i will run into situations where i cant do vlans and would need more physical nics.
 

marcoi

Well-Known Member
Apr 6, 2013
1,532
288
83
Gotha Florida
Build Out Decided. I went with the following combo.

Motherboard: ASrock E3C246D4U
CPU: Intel Xeon E 2176G
RAM: 4x samsung 32gb ddr4 ECC ram
CASE: Re-using Node 804
AIO CPU Cooler: Fractal Design Celsius S24 Blackout 240mm Silent High Performance
HDD: Reuse lsi raid card with 1GB Cache and BBU with 4 or 8 250GB SSD or just 6 SSD on using Motherboard.
NIC: Reuse intel 4 port card.
NIC2: Reuse 10GB SFP+ card
PSU: EVGA 1000W PSU I bought on sale last year.

Build Purpose:
I want to build a new server to run multiple instances of Router/Firewall VMs. The purpose is to have all the networking and a few other VMs running on the box so when i need to bring down a server for maintenance i dont lose networking.

VMs so Far
  1. ATT PFsense - I replaced my gigafiber router with pfsense VM recently. This VM connects my network to ATT service and provides WAN in. It also provides subnet for IOT wireless AP.
  2. Sophos UTM9 - I use this for home network to reduce/control/block internet garbage. I have a lot of rules setup so I dont plan on replacing it with something else anytime soon.
  3. PFSense_External1-4: I want to setup between 1-4 pfsense vms for each of my external IP adddress. I have 5 available. Each PFsense will deal with it's own DHCP and firewall rules for services running on that IP.
  4. Backup WS2012R2: Provide home network DNS
  5. WS2012R2 Essentials: Provides main DNS and backup/storage access. Storage is iSCSI attached back to freenas vm.
  6. W10 tester: Just to access various Pfsense images.
Setup Plans:
Once i have the system built out and tested. I plan on installing ESXI on an internal usb thumb drive.
Depending on if i use the raid card or motherboard sata, i will setup either a raid 10 8 drive storage using the 8 x 256GB SSD Drives or i will setup 6 drives and have individual storage for each VM.
I will setup the intel 4port nic and 2 onboard nics as passthrough devices so pfsense VMs can use it.
The 10GB SFP+ will be used for storage access.
If I need more nic ports, i will replace the raid card with 2nd intel 4p nic.
I dont know if the CPU GPU can be setup as passthrough, if so i might add it to vm.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I think you will need to do SRIOV if you want to pass one card to multiple VMs, else passthrough is a 1 card : 1vm relationship

O/C the card and board need to support that and need to be configured appropriately (bios and potentially cli)
 

marcoi

Well-Known Member
Apr 6, 2013
1,532
288
83
Gotha Florida
I think you will need to do SRIOV if you want to pass one card to multiple VMs, else passthrough is a 1 card : 1vm relationship

O/C the card and board need to support that and need to be configured appropriately (bios and potentially cli)
That's true. I dont think i want to use SRIOV. I should be ok with 4 port nic and 2 onboard. I will setup the 4 port nic as pass though in ESXI, which will shows up as 4 individual nics that i can assign to various VMs.

Break out as follows:
--ESXI Level - Create a vswitch group call VmNetwork and assign 2 onboard nic - connected to the switch
--ESXI Level - Create a vswitch group call Storage and assign 1 10GB nic - connected to the switch on 10gb
--ESXI Level - Create a vswitch group call ExtNetwork and assign 1 physical nic - connected to the switch
--ESXI Level - Create a vswitch group call LAN and assign no nics - Local vswitch only.
--ESXI Level - Create a vswitch group call StaticIP and assign no nics - Local vswitch only.

1. ATT PFsense
Interface 1 WAN - Assign 1 phy nic for ATT ONT in.
Interface 2 -LAN - Assign virtual nic from LAN group.
Interface 3 -IOT - Assign 1 phy nic connected to pswitch vlan.
Interface 4 -StaticIP - Assign virtual nic from StaticIP group.

2. SophosUTM
Interface1 WAN - Assign virtual nic from StaticIP group.
Interface2 LAN- Assign virtual nic from VmNetwork group.

3. PFsense Ext1 VM
Interface1 WAN - Assign virtual nic from StaticIP group.
Interface2 LAN- Assign virtual nic from ExtNetwork group

4. PFsense Ext2 VM
Interface1 WAN - Assign virtual nic from StaticIP group.
Interface2 LAN- Assign virtual nic from ExtNetwork group

5. PFsense Ext3 VM
Interface1 WAN - Assign virtual nic from StaticIP group.
Interface2 LAN- Assign virtual nic from ExtNetwork group

6. PFsense Ext4 VM
Interface1 WAN - Assign virtual nic from StaticIP group.
Interface2 LAN- Assign virtual nic from ExtNetwork group

The Pfsense Ext1-4 will need to use vlans to separate the traffic out to Other VMs using those FW.
 

marcoi

Well-Known Member
Apr 6, 2013
1,532
288
83
Gotha Florida
Build Specs so far:
Motherboard: ASrock E3C246D4U
CPU: Intel Xeon E 2176G
RAM: 4x samsung 32gb ddr4 ECC ram
CASE: Re-using Node 804
AIO CPU Cooler: Fractal Design Celsius S24 Blackout 240mm Silent High Performance
HDD: 4x Samsung Evo 850 250GB SSD and 1x Samsung Evo 850 500GB SSD
NIC: Reuse intel i350 4 port card.
NIC2: Reuse 10GB SFP+ card
PSU: EVGA 1000W PSU I bought on sale last year.

with ESXI installed on usb and all 9 fans set to 100% the box idles at 55 watts.

Im still working out the deployment to move my VMs to the server. Hopefully i can do it over the weekend.

Here are some pictures of the build.
20190918_210246.jpg 20190918_210103.jpg 20190918_205834.jpg 20190918_205830.jpg 20190917_232243.jpg 20190917_232234.jpg 20190917_221615.jpg

I really like the AsRock IPMI interface. Also the Fans at full speed are "silent" against my other servers lol.
The AIO keeps the CPU nice cooled.
Im still waiting on bios update to make use of the full 128GB Ram.
I also got the intel GPU pass-through. But there isnt an output option so i havent decided how to use it. Might be used for transcoding like blue iris software for cameras.
upload_2019-9-20_15-54-52.png
 
Last edited:
  • Like
Reactions: Tha_14

marcoi

Well-Known Member
Apr 6, 2013
1,532
288
83
Gotha Florida
new box is in production.
I havent spent much time doign testing but today i was downloading Steam game and noticed highest download speed ever
upload_2019-9-26_14-1-42.png

looks like my sophos utm is making use of the new cpu :eek:
upload_2019-9-26_14-14-43.png
 
Last edited: