Any (free) Hypervisors That Can Bridge NICs In The VSwitch?

[EasyGoing1]

Hardware: 4 port 2.5G on-board NICS (Intel i225V) - fanless box with an N5015 CPU, 32 Gigs RAM, 1TB SSD Storage.

Firewall: OPNSense (virtualized)

Desired Config: Three ports are assigned to a single bridge. Each virtual NIC is connected to that bridge, including the management vNet. The remaining physical NIC is dedicated to OPNSense via PCI passthru. Then a single virtual NIC is given to OPNSense so that OPNSense thinks it only has two NICs, one for the WAN and one for the LAN (the LAN now existing on a three-port bridge that OPNSense knows nothing about).




ESX

I've wrestled with this configuration off and on for some time. I first tried it with ESXi 8 but without paying for vSphere, this config is a NO GO. However, I find it interesting that they only make this config possible if you purchase vSphere, they even describe it in their documentation and point out that it requires a purchased license before it can be done.

What I have been doing with ESX to have some functionality, is I gave OPNSense three of the four NICs via PCI passthru, and I bridged two of those NICs in OPNSense (you can't bridge virtual NICs in OPNSense because bridging can only happen when the OS has access to the hardware directly via the kernel). So the final NIC count for OPNSense was FOUR NICs - THREE via passthru and one virtual NIC. So then OPNSense has essentially three isolated NICs to work with (WAN, Virtual, Bridge) where my main private LAN is on the bridge. The virtual nic is used only for managing ESX when I need to work with it while the OPNSense virtual machine is shut down (that need arises from time to time). The game console stays on WiFi since I can't dedicate a port to it ... yeah, I could share it with the port that also has the virtual network on it, swapping out cables when I need to use that port for ESX management, but this is not the optimal config and not the config I want to settle on. This would also take the game console off of my main LAN which is again, not desirable.

With ESXi, if you try to add NICs to NIC Teaming, it only provides settings that are valuable in a fail-over scenario. It requires a "Load Balancing" setting which tells it how to ROUTE packets from the physical NIC to the upstream connection, where I am assuming it only allows traffic from the virtual NICs to traverse one NIC and if that NIC fails or goes offline, it then fails over based on that setting. AND, if I add three NICs into the team and try setting them all to Enabled, the ESX box becomes 100% unreachable and the only way I can recover it is to wipe the entire config at the console. Wiping the network and resetting it to defaults won't work, I have to wipe the entire config, then re-import the virtual machines manually because they just vanish (though their files are all still there).

PROXMOX

I recently gave it a go with Proxmox, and though it was possible, it did it only when I created the bridge at the Debian level first. So then Proxmox is just handed a virtual NIC from the OS, making it none the wiser that it's virtualizing a virtualized NIC (not that this should matter at all). However, the performance of the virtualized firewall was horrid, giving me a thruput of less than 100 megabits on my gigabit Internet service (I got full gigabit Internet speeds with the ESX setup - same hardware, same cables, etc.). I'm not sure whether the issue with the performance is a Proxmox issue or a Debian / NIC driver issue - I lean on the OS as being the most likely cause of the bottleneck, and so I am looking into that at the moment, but my frustration levels are maxing out with this setup.

HYPER-V

I was reading up on Microsoft Hyper-V today, but the language they use when describing "teaming" NICs into a virtual switch - is eerily similar to what I've seen with ESXi where it seems to be a fail-over only scenario - though I haven't dug into it much deeper than that, especially after I read that their implementation of SR-IOV only works with Windows virtual machines ... this, of course, makes sense since Microsoft's universe doesn't know that Linux exists (with a few rogue exceptions within their dev dungeons).

What I am hoping to find from this post, is someone who is aware that this config can be done in some bare metal hypervisor that exists somewhere on the Internet that I can download and test.

Thank you,

Mike

 

[DavidRa]

I don't get it. What is the bridge giving you? As a general rule the point of a bridge is to connect two different layer two networks - but I don't understand why switching won't fulfil the same need.

Basically I'm suggesting this is an "XY problem" and you're asking for how to make a solution work instead of asking how to solve the problem.

Edit: Wait. I think I figured it out. You want to use the three ports in your $500 firewall/server as a switch instead of getting an 8W 5 port switch and doing it ... Normally?

 

[RTM]

There's a bit too much information in the original post, that isn't really required. Especially it seems there is some confusion as to what "teaming" is and when to use it.

OP, teaming is not what you want, it is for creating a logical link over more than one cable (to get redundancy and/or extra bandwidth).

But anyway, my understanding is that the OP wants to create a network, where VM's and physical hosts are able to communicate more or less directly with each other (same L2 broadcast domain).

It's been a while since I've used ESXi, so I may be a bit rusty, but if I remember correctly you just need to add the interfaces to the same port group (where you also add the interfaces of the VM's). You may need to enable/disable some features on the port group.

Unless things have changed with ESXi 8, you should be able to do it without getting a paid license.

I haven't worked (much) with hyperv or proxmox, so I couldn't tell you how to do this, but I assume it should be more or less the same.

 

[DavidRa]

Yeah OP seems to want the virtual switch "bridge" to act like a real switch for all the connected devices. I don't think that's going to happen on any platform - or at least not one I know.

Could spend days trying to make this work but it's not the right way to do it (if I understand the desire at least). OP - get a proper network switch and do it the simple way. There's a reason this is "how it's normally done".

 

[ScaRuleZ]

If your need is to create a switch, why don’t you just pass the other nics to OPNSense and let it manage the ports?

 

[BoredSysadmin]

I think you messed up the network config on ESXi, as this is 100% possible to run PFsense and Opnsense on free ESXi (without vCenter)
here is the guide (for PFsense, but it should be easy to adjust it for Opnsense)
pfSense® software Configuration Recipes — Virtualizing pfSense Software with VMware vSphere / ESXi | pfSense Documentation (netgate.com)

Basically, you create TWO standard vSwitches WAN and LAN. Pass virtual nic ports from both to Opnsense and let it bridge/firewall etc..
Then connect your internal VMs to LAN vswitch.

 

[EasyGoing1]

Yeah OP seems to want the virtual switch "bridge" to act like a real switch for all the connected devices. I don't think that's going to happen on any platform - or at least not one I know.

Could spend days trying to make this work but it's not the right way to do it (if I understand the desire at least). OP - get a proper network switch and do it the simple way. There's a reason this is "how it's normally done".

As I stated, I did get it working in Proxmox but the performance was not acceptable. And yes, you are correct, I want to use these ports as a switch as one would do with any firewall / router that has a WAN interface and a group of ports dedicated to the private LAN. Sometimes, the academics of a problem can make the problem worthy of solving.

 

[EasyGoing1]

I think you messed up the network config on ESXi, as this is 100% possible to run PFsense and Opnsense on free ESXi (without vCenter)
here is the guide (for PFsense, but it should be easy to adjust it for Opnsense)
pfSense® software Configuration Recipes — Virtualizing pfSense Software with VMware vSphere / ESXi | pfSense Documentation (netgate.com)

Basically, you create TWO standard vSwitches WAN and LAN. Pass virtual nic ports from both to Opnsense and let it bridge/firewall etc..
Then connect your internal VMs to LAN vswitch.

I've been running OPNSense on ESXi for over a year now, I know it's possible. What isn't possible is creating a bridge that is visible to VMs and physically connected devices on the same subnet. OPNSense cannot bridge multiple virtual NICs and the free version of ESXi will not allow you to create a multi-port bridge where all ports in the bridge are live and working and can communicate with all VMs and other physically connected devices.

Handing the physical NICs over to OPNSense to manage via PCI passthru removes those NICs from ESXi' which means that virtual machines cannot communicate with any of the devices connected to the OPNSense managed bridge.

I need the hypervisor to create and manage the bridge so that both physically connected devices and virtual machines all share the same layer 2 domain.

 

[BoredSysadmin]

I've been running OPNSense on ESXi for over a year now, I know it's possible. What isn't possible is creating a bridge that is visible to VMs and physically connected devices on the same subnet. OPNSense cannot bridge multiple virtual NICs and the free version of ESXi will not allow you to create a multi-port bridge where all ports in the bridge are live and working and can communicate with all VMs and other physically connected devices.

Handing the physical NICs over to OPNSense to manage via PCI passthru removes those NICs from ESXi' which means that virtual machines cannot communicate with any of the devices connected to the OPNSense managed bridge.

I need the hypervisor to create and manage the bridge so that both physically connected devices and virtual machines all share the same layer 2 domain.

I don't understand the issue of "bridging" multiple interfaces. Are you trying to get some sort of nic teaming or LACP?

Again, if you configure a second vSwitch for the LAN network and assign Physical interfaces as uplinks and virtual interfaces in port groups, I don't see why both virtual and physical devices couldn't use the same network. Think of standard vSwitch as a basic layer two switch.
Why OPNsense won't be able to bridge WAN and LAN if both provided a vNics - you'll get presented with two valid interfaces in Opnsense. Just take note of MAC addresses to identify which one is WAN and LAN.

Here's another pfsense/ESXi guide - you can follow the 2 nics steps:

Setup Pfsense In Virtual Machine (Vmware ESXI) | Coy Geek

Download Pfsense

coygeek.com

 

[marcoi]

IDK if it worth it for you, but you can get vcenter with VMUG membership for 200 a year. It give you access to alot of other VMware software. With vcenter you can do the virtual switch etc.

 

[BoredSysadmin]

IDK if it worth it for you, but you can get vcenter with VMUG membership for 200 a year. It give you access to alot of other VMware software. With vcenter you can do the virtual switch etc.

sure, VMUG isn't that expensive. I pay 120/year for Grammarly. But my point is that vCenter and vDS are not needed here unless I misunderstand and OP IS interested in port channel/LACP (which isn't available in standard vSwitch, but requires vSphere Distributed Switch), then YES, you'd need enterprise plus or vSphere+ (subscription) license or VMUG.
btw: ProxMox (free) does support link aggregation (aka LACP) if this is what OP needs:

Network Configuration - Proxmox VE

pve.proxmox.com

 

[DavidRa]

OP already confirmed that they want multiple ports on the host (most likely on independent network controllers) to act like a real switch as well as a vSwitch. And yes, while "hardware firewalls" do something like this they're not also trying to run through a hypervisor and attach a set of VMs - so they can just bridge them in the firewall OS if they're not already on a special switching controller.

OP's different requirement seems to be that the hypervisor needs to configure the ports as a real switch as well as attaching vNICs - but AFAIK the products just aren't designed or built to do that. They even admitted that the one time it sort of worked on Proxmox, performance was poor.

References:
* ESXI single vSwitch with 2 physical NICS
* https://docs.vmware.com/en/VMware-v...UID-DFE769C9-9A9C-4CDB-A0BC-2B17931B75A5.html - note that the multiple adapters "appear in the failover group list under the Assigned Adapters list"

Honestly - it's not even an interesting academic problem to me (there's learning new ways to do things, and then there's asking why the 32 bicycles can't pull the caravan) so I'm going to bow out.