How do I PROPERLY configure an N5015 device with ESXi and OPNSense?

[EasyGoing1]

Hello,

So I bought from Ali Express, one of the N5105 fanless boxes - bare bones model then I got 32 gigs of RAM and a 1TB SSD from Amazon - total cost out the door less than $300.

I've installed ESXi as the booting OS, then I installed OPNSense in a VM, and I have it properly working and routing packets from the LAN to the WAN and that's great ... but what I want to do is configure three of the four ports to act like a switch where all three ports are on the LAN. Anyway, my issue seems to be either with ESXi and not understanding it fully, or it's an issue with my knowledge of OPNSense (which I've never used before until a couple of days ago) and not knowing how to properly configure bridging. So let me first explain how I have this set up so you have a good picture of my environment.

Within ESXi, I have FOUR vSwitches. Each vSwitch (0 - 3) is assigned to each port on the main board. However, since the Management network likes to default to being on vSwitch0, I assigned vmnic1 to vSwitch0 and vmnic0 to vSwitch1 ... so here are those assignments:

vSwitch0 - vmnic1
vSwitch1 - vmnic0
vSwitch2 - vmnic2
vSwitch3 - vmnic3

THEN, I have these Port Groups:

WAN - vSwitch1
LAN_1 - vSwitch0
LAN_2 - vSwitch2
LAN_3 - vSwitch3

Then, I created the virtual machine for OPNSense and I assigned it four NICs, which are each of the Port Groups, one per virtual NIC. I hard-coded the MAC addresses of the VMs virtual NICs so that the last octet was numbered sequentially which made it easier to connect which Port Group was which NIC inside OpnSense (the lowest number would be on port group WAN, and the highest would be on port group LAN_3 with the lowest numbered MAC address being vmnic0 and connected directly to the cable modem).

When I got OPNSense installed and did the manual assignment of network interfaces, I referenced the mac address that I assigned to each nic in the VM settings and was able to assign the proper port to the WAN interface in OPNSense, then the next mac address went to the LAN, then other two ended up being labeled as OPT1 and OPT2.

So far it seems fairly straightforward to me and all I should need to do at this point is pool the LAN interfaces into a bridge. But this is where I'm having problems.

For example, when I enable both OPT1 and OPT2 without giving them an IP address, then I go into Interfaces under Optional Types and create a bridge and assign those two interfaces to the bridge, then go into assignments and create a new interface that is assigned to the bridge, then I enable the new interface that I created, and give the bridge interface an IP address (on a different subnet for testing purposes only), then I connect a physical NIC from my laptop directly to either of those bridged ports, I cannot ping that interface to save my life.

I've tried everything I can think of and I even looked over and tried to implement this guide To no avail.

I'm somewhat at a loss here, because what I really need to do is have those three ports all on the LAN interface in OPNSense but I can't seem to make it work, and I'm hoping someone will have some insight for me to help me get this dialed in.

Thank you,

Mike

 

[itronin]

reading this made my head spin just a bit with how you are trying to accomplish vs what you've stated you want to do. Maybe there's more you are thinking about down the road? IDK.

so my question: why not just pass the nic hardware through to opnsense (well except for the mgmt interface consumed by vmware)???

You can always send any other vm's network traffic out the mgmt interface port group and if you really need to get fancy you *can* vlan that if you want to segregate traffic - not saying you should mind you... but you can.

Sounds to me like you are only wanting to treat those other 3 ports as an actual switch (bridge group). be best I think to let opnsense manage that rather than insert meta layers in between the hardware and opnsense (your esxi vswitch / port groups) - adding those layers also means you are adding additional latency and cpu processing by esxi on top of what you want opnsense to do...

while costing a bit more - a simple unmanaged 2.5gbe switch would also be relatively clean and give you the flexibility of taking specific vm's and passing their traffic through specific ports - which vmware is good at...

 

[EasyGoing1]

why not just pass the nic hardware through to opnsense (well except for the mgmt interface consumed by vmware)???

This is what I've done. By making the vSwitches and Port Groups, I was able to pass all four NICs on to OPNSense without any issues.

You can always send any other vm's network traffic out the mgmt interface port group and if you really need to get fancy you *can* vlan that if you want to segregate traffic - not saying you should mind you... but you can.

I shouldn't have to get into VLANing in ESX to do this ... though I thought I could make my bridge using a single vSwitch on the ESX side, which I did try, by assigning the three NICs as uplink ports on the vSwitch, then I assigned the vSwitch to a Port Group and then I assigned the port group to the OPNSense virtual machine. That worked, but as soon as plugged in a second device onto the bridge, the first device that I had plugged into it ... dropped off the network and it then favored only the second device that I plugged in.

Sounds to me like you are only wanting to treat those other 3 ports as an actual switch (bridge group). be best I think to let opnsense manage that

This is exactly what I want to accomplish. However, try as I might, I cannot seem to get OPNSense to create a WORKING brige port of any kind.

What is odd is that when I do create a bridge port from only two of the NICs (so then One would be WAN port, another would be LAN port and then the Bridge port with the remaining two NICs), I can ping the bridge port from behind the LAN interface, but when I plug my laptop directly into any of those two ports that make up the bridge, they won't get an IP address and if I hard code an IP I can't ping the IP address of the bridge interface. I can only ping it from the LAN interface. Very strange behavior.

while costing a bit more - a simple unmanaged 2.5gbe switch would also be relatively clean and give you the flexibility of taking specific vm's and passing their traffic through specific ports - which vmware is good at...

I've already been considering this, but I cant get past the fact that this should be doable ... just need to figure out what I'm doing wrong, or find someone who's configured this same hardware before in the manner I'm trying to ... who was successful then mimic their config.

Thanks for taking the time to respond.

Mike

 

[itronin]

This is what I've done. By making the vSwitches and Port Groups, I was able to pass all four NICs on to OPNSense without any issues

Mike

sorry for the confusion and I don't mean to sound harsh. But no, you aren't passing through the hardware, at least not in the literal esxi sense. you are dedicating vswitches and port groups to a guest OS. These two concepts are not the same.

by "pass through" I meant hardware pass-through of the pcie devices. technically speaking, creating vswitches, and port groups is not passing the nic hardware through. ESXI still manages those devices, still manages, massages, and moves the data across nic interfaces - in addition to opnsense doing it. Hardware pass through literally means the guest OS (opsense in this) has direct access to the underlying hardware. its basically the same as if you had bare metal install of an os on a PC...

this is the same as if you passed thorugh an HBA or a GPU to a guest OS.

 

[EasyGoing1]

sorry for the confusion and I don't mean to sound harsh. But no, you aren't passing through the hardware, at least not in the literal esxi sense. you are dedicating vswitches and port groups to a guest OS. These two concepts are not the same.

by "pass through" I meant hardware pass-through of the pcie devices. technically speaking, creating vswitches, and port groups is not passing the nic hardware through. ESXI still manages those devices, still manages, massages, and moves the data across nic interfaces - in addition to opnsense doing it. Hardware pass through literally means the guest OS (opsense in this) has direct access to the underlying hardware. its basically the same as if you had bare metal install of an os on a PC...

this is the same as if you passed thorugh an HBA or a GPU to a guest OS.

So I was able to get it working using passthru as you suggested, but that ultimately defeats the config I was hoping to achieve where I could also share those NICs with other VMs on the host... I decided instead to just have the NICs provided to OPNSense as virtual NICs where I assign a different subnet to each NIC. It's not ideal, but it works and as I implement and use local DNS resolution, the different subnet addresses won't even be a second thought.

Thanks for your time.