Virtualizing TNSR alongside VMs on ESXi 7

takeawaydave

Member
Aug 20, 2013
62
2
8
Software company I work for are starting to check how secure home labs are. Good idea but means I need to get a few things redesigned.

I need to build a more secure zone in my home network in to which I can deploy and run various VM’s.. I was planning to convert my home network to 10GBE (mainly between office (where I work) and basement (where I have server, nas, switch)

Looking on netgate.com an adequate appliance with 10GB throughput seems to be based on an Intel Xeon Processor D-1537 - a low TDP class 8 CPU (Intel Xeon CPU D-1537 @ 1.70 GHz) based machine.

Plan is to virtualize TNSR on to ESXi 7 running on PowerEdge 630 which is currently running with 16 CPUs x Intel(R) Xeon(R) CPU E5-2630L v3 @ 1.80GHz.

I have a a double X520-DA 10GB card - one of the ports is used for a 10 GBE DAC connection to a Synology NAS (this provisions ESXi datastore over iSCSI)

Could the remaining 10GB port be passed through directly to the TNSR VM with a 10 GBE uplink to suitable 10 GBE switch (Probably USW-Agregator) ?

On the internal side then set up a new ESXi vSwitch with multiple VLAN’ed port groups in order to connect to the VM subnet. No additional physical adapter needed, right ?

Any suggestions or advice on best way to get going ? I’ll admit I’m not too hot on network security nor networking in general since it’s not my day to day work but not a complete noobie either…
 
Last edited:

Vesalius

Active Member
Nov 25, 2019
215
155
43
I haven't used ESXI much so I'm not sure how it handles direct passthrough of PCI devices, but as far I have found on proxmox with my hardware Direct I/O passthrough would send the whole PCI card to a VM and you can't split the ports up. SR-IOV would allow you to use the single PCI card and passthrough multiple virtual functions (up to 64 for the x520) of that NIC to many different VM's. So one VF could continue being used by your NAS and the other go-to TNSR and still others could go to other whatever VM you choose to test next.
 
Last edited:
  • Like
Reactions: takeawaydave

RTM

Well-Known Member
Jan 26, 2014
877
330
63
I can not recommend virtualizing routers/firewalls.
Depending on how your network is designed, you may have trouble accessing the hypervisor if the firewall is offline (perhaps due to an issue with the hypervisor).
Another example, if you need to update the hypervisor, you will need to put it into maintenance mode, which means your network VM will be offline, but if you can't access the hypervisor without the VM running... you get it..

There are other reasons, and ways to get around the issues, but again I think for a home network you are better off not virtualizing.
 
  • Like
Reactions: takeawaydave

Vesalius

Active Member
Nov 25, 2019
215
155
43
I can not recommend virtualizing routers/firewalls.
Depending on how your network is designed, you may have trouble accessing the hypervisor if the firewall is offline (perhaps due to an issue with the hypervisor).
Another example, if you need to update the hypervisor, you will need to put it into maintenance mode, which means your network VM will be offline, but if you can't access the hypervisor without the VM running... you get it..

There are other reasons, and ways to get around the issues, but again I think for a home network you are better off not virtualizing.
Reading the OP this is an ideal use case for virtualizing routers/firewalls.

beyond that fear of virtualizing a router/firewall even as the edge firewall in a home lab (which is not the OP’s use case) is greatly overblown, my opinion obviously, if both the host hypervisor and the firewall/router are treated as production environments and not let’s play/tinker and reboot that node every couple of minutes.
 

RTM

Well-Known Member
Jan 26, 2014
877
330
63
beyond that fear of virtualizing a router/firewall even as the edge firewall in a home lab (which is not the OP’s use case)
Obviously I disagree about virtualizing, having done it myself a number of times.
How are you sure that it will not be the edge firewall? I do not see that level of detail.

There is some additional context by the way:

In any case, I think it would be a good idea to try to describe what you (OP) are trying to build, perhaps make a network diagram (draw.io is a good place to do this).
 

Vesalius

Active Member
Nov 25, 2019
215
155
43
Obviously I disagree about virtualizing, having done it myself a number of times.
How are you sure that it will not be the edge firewall? I do not see that level of detail.

There is some additional context by the way:

In any case, I think it would be a good idea to try to describe what you (OP) are trying to build, perhaps make a network diagram (draw.io is a good place to do this).
My (maybe bad) assumption that the testing of homelab security was about testing various router/firewalls virtually. I apologize because I worded that too strongly.

agree completely on getting more network info on exactly what the OP is trying to do and what hardware he has to do it with.

no issues with aggreeing to disagree on the virtualized router/firewall topic. I have done so for many years with my home business and personal home network. Many enterprise solutions run that way as well.
 

takeawaydave

Member
Aug 20, 2013
62
2
8
network_current.png

Thanks @RTM to draw.io - above is the current network setup.

I need to introduce a secure zone as illustrated below - apologies @Vesalius if I wasn’t clear upfront.

network_future1.png

The laptop icon Is my work issued laptop.

All connections are 1 GBE apart from the DAC connection which runs between two X520-DA cards - in ESXi (Dell PowerEdge R630) and Synology NAS.

I was also planning in the next few months to upgrade to 10 GBE from ISP and run a 10 GBE network backbone.

At the moment the network devices are all Ubiqiti and conversion to 10 GBE was planned to be done using Dream Machine Pro and USW Aggregator Switch.

I am getting unclear now on how I could define a firewall to define the Secure Zone.
 
  • Like
Reactions: RTM

RTM

Well-Known Member
Jan 26, 2014
877
330
63
So... if I understand you correctly, the TNSR VM will be the gate for the traffic to/from HOME ZONE to SECURE ZONE?

If that is the case, then it would not be a big problem to virtualize it, as you will not lose internet access from your home zone if it does not work.

I assume part of the "SECURE ZONE" concept is to separate the lab from the HOME ZONE as much as possible, and only the work laptop should have access to the lab, as such I suggest the following:

In your home zone (if you haven't already), segment the various devices into different subnets, such as:
  • 10.0.0.0/24 - home devices
  • 10.0.1.0/24 - work laptop
  • 10.0.2.0/24 - NAS
This will make it possible to configure a rule in TNSR, that will allow the work laptop (and only that) access to your lab vlans (where I assume you have various services on).

While you are at it, you should consider how you will access management of your lab environment, specifically the hypervisor and iDRAC (will probably mean you need to use a additional 1G interface), will that be through the TNSR VM? (doing so would help separate the lab environment, but also makes it a bit of a hassle if TNSR is down).

Furthermore, as your NAS appears to have a dual role, in my opinion it breaks with the HOME/SECURE ZONE concept, you may want to consider not using it for your lab or home entirely (if that is even an option).
If you do some segmenting as mentioned above, someone could possibly pivot from "home devices" to NAS to secure zone.
 

takeawaydave

Member
Aug 20, 2013
62
2
8
Yes, @RTM - The TNSR VM would be the gateway for traffic to/from HOME ZONE to SECURE ZONE - Yes, the assumption is to ensure only work laptop can access the VM’s in the secure zone.

Currently, I have a main HOME subnet alongside multiple WORK VLAN’s configured on the USG (later plan the same on the UDM-Pro) however not quite yet down the lines how is proposed above which I think make sense and would be a good start for building out.

As far as I understand defining 3 separate Subnets; HOME, WORK LAPTOP, NAS could be done.

Reading the following:

https://www.reddit.com/r/PFSENSE/comments/b94dr3/_/ek5acij
If VLAN definition remained on USG/UDP-Pro could CE pfSense be used instead of TNSR Up to a throughput rate of 9.5 Gbps ? I guess this would however mean throwing more cores at pfSense compared to what TNSR needs and going bare metal ?

The reason I ask is because I read that the TNSR Home Lab edition is not updatable which is a pretty big negative. Basically looking for the 10 Gbps through put from Work Laptop to WORK SECURE avoiding TNSR and using pfSense in conjunction with UDM-Pro - is this possible?

Apologies if that‘s an annoying question but hopefully highlights some of my gaps in understanding :)

Yes, the NAS is dual function - backup target for some home devices - ESXi datastore over iSCSI for ESXi in SECURE ZONE. Technically an attack vector however could live with the risk since the device is adequately hardened and not internet facing.

@Vesalius - No, secure zone VM‘s would not have direct access to the NAS in the sense of having access to shares. Only their virtual disks would be residing on NAS hosted datastore.
 

Vesalius

Active Member
Nov 25, 2019
215
155
43
I think that your proposed use for the NAS should be secure. The insecure VM's can be set up to not have any network connection to the NAS, they will only see the storage space presented to them as a disk from ESXi.

So the USG/UDP-Pro will remain your edge router connected to external WAN/internet. Are any of your current Work USG/UDP-Pro VLANs secure/segregated enough to allow the ESXi router/firewall VM to use it as WAN/external now? If so, then there is one of your answers. If not, it is probably cleanest to create a new VLAN on the USG/UDP-Pro and USW-8 tagged and USW-24 (untagged using brocade terminology here) on the chain of ports between the USG/UDP-Pro to the physical ESXi nic port. This will only serve as the internet-facing wan/egress for the ESXi router/firewall VM

There are many ways to skin this cat, one way is below. I treat the ESXi router/firewall VM like it was a router on a stick (10g stick). So the single 10 g connection would carry the ESXi router/firewall VM internet-facing WAN, whatever current VLAN your WORK LAPTOP currently resides on back to the ESXi router/firewall as well as whatever VLANS currently are used between the ESXi host and your network.

on the USG/UDP-Pro
  1. create a new VLAN (267 for example) and subnet that only has access out to WAN. Use the interface/nic connected to the USW-8
on the USW-8
  1. Create VLAN 267
  2. Tag VLAN 267 on the ports connecting the USG/UDP-Pro and the USW-24
on the USW-24
  1. create VLAN 267
  2. tag VLAN 267 on the port connecting the USW-8
  3. on the port connecting to ESXi
    1. untag VLAN 267
    2. tag your WORK LAPTOP VLAN
    3. Tag whatever other VLANs you have for the ESXi host or secure current VM's

In ESXi you give the router/firewall VM:
  1. a SR-IOV vf nic
  2. at least one virtual nic (vlan aware) or a second SR-IOV VF EDIT: thinking about it some more I don’t think the 2nd SR-IOV VF will be a sharable option among the EXSi router/firewall vm and the insecure VM‘S. Likely needs to be an EXSi virtual nic or virtual switch to be shared among them all. I use Proxmox so I am unsure of EXSi’s exact terminology and features on this.
in ESXi you give the insecure Work VMs only the Virtual nic given to the EXSi router/firewall VM

then at the ESXi router/firewall VM level you define:
  1. WAN/Egress out the allocated SR-IOV vf interface without a VLAN. This will be tagged with VLAN 267 at the USW-24
  2. the WORK LAPTOP VLAN using the allocated first SR-IOV vf interface.
  3. VLANs for the insecure ESXi VM's using the Virtual NIC or second SR-IOV VF interface.
  4. define routing rules that allow intervlan communication between your WORKPLACE laptop in #2 and the insecure VLAN's' in #3 above.
Both Pfsense and OPNsense can likely do well enough at 10g, if you were willing to do CLI with TNSR then VyOS is an option you might consider for the ESXi router/firewall VM as well. VyOS is overall faster than either of the *sense's.
 
Last edited:
  • Like
Reactions: takeawaydave

takeawaydave

Member
Aug 20, 2013
62
2
8
Thanks @Vesalius for the details earlier in the week. Looking to try and get this done over the weekend. Will report back.

** Having plenty of issues around passing sr-iov vf from the quad i350 pf to esxi **

- might need to settle on setting up a vSwitch for the WAN side instead I suppose.
 
Last edited: