ESXi 7 - VLAN passthrough / Strange vNIC assignement behavior

Partofthesolution

New Member
Dec 3, 2020
28
7
3
Dear forum,

I am running an ESXi7 installation (latest update 7.0.1c is installed).

My first topic is:

I installed a opnsense router/firewall in a guest VM. First, i tried to config this guest with three virtual NIC/vmxnet 3 adapters.
The first vNIC is linked to a physical 1Gbit-Intel-NIC for WAN / ISP-connection.
The second vNIC linked to a physical 10Gbit-Intel-NIC for connections to the local physical nerwork through a manageable switch.
The third (and possible the fourth an so one) is linked to virtual networks only.

The first topic refers to the second vNIC:
I tried to configure VLANs inside of the guest, which means i added VLAN interfaces in opnsense. My problem is: It wasnt possible to get any ethernetframes with VLAN-ID from or to the physical switch. So the question is: How can i passthrough 802.1Q from a guest OS to the physical world? What do i have to configure?

Managing VLANs inside of the ESXi-hypervisor with varying number of vmxnet adapters is not an option for two reasons:
1. Reason: Its not possible to add more than 10 vmxnet 3 adapters to a guest, but i will have more than 10 VLANs. not now, but in a future szenario.
2. The config should be export-/importable to use it on a physical PC Engines APU 4 for fall back scenario without the (or less) need of reconfiguration any devices/interfaces.


Second topic:

The solution of this second topic will be obsolet if there is a answer for the first topic, but i also would like to understand whats going on with this issue.

To finally start with the project, it is ok for me to add up to ten vmxnet 3 adapters for now. So i initially started with the following config to get access to the opnsense webui and to the internet.

vmx0 = WAN
vmx1 = LAN
vmx2 = WiFi private
vmx3 = WiFi guest

I can confirm this assignment by downloading the opnsense config.xml

1609603395464.png

After i confugured basic settings, i shuted down the opnsense and added two more vmxnet 3 adapters, but it wasnt possible to reach the opnsense after that (no matter if i check the "Connect" option or not).

vmx4 = WiFi smart devices
vmx5 = Wifi HomeOffice devices

1609603451067.png

After removing Network Adapter 5 and 6, it was possible to reach opnsense again. I tried to figure out the reason for this issue:

Console window of opsense showed, that all assigned interfaces were listet after boot no matter i boot wíth four or six vNICs. Also, vm0 still established a PPPoE-connection to my ISP, so in general, opnsense was running, but for some mysterious reasons, it wasnt possible to reach the internal webui and ssh anymore or ping it.

While the opnsense was running with six adapters, i removed Network Adapter 5 and 6. I expected, that vmx4 and vmx5 would have detached, but the result was, that vmx1 and vmx3 was detached (which intially was assigned to Network Adapter 2 and 4!):

1609604235625.png

After that, i started opnsense with only the intial four adapters, so opnsense was available again. I added the fifth and sixth adapter while the guest was running and in the console windows, vmx4 and vmx5 was attached.

1609604349330.png

Right after adding the fifth and sixth adapter, i still reached the webui and it was possible to assign the devices. After the next reboot, opnsense wasnt available anymore.

I also noticed, that clients got DHCP leases from different ranges. It seems, that vmx# assignement gets shuffled after adding new vNICs to a guest in a unknown way.

Whats the reason for this issue?
 

Rand__

Well-Known Member
Mar 6, 2014
5,381
1,147
113
I am only running 6.7, but I dont think much has changed.

Re topic 2 - it sounds as if the order of NICs your vm sees is not identical to the order you see in the gui.

Re topic 1
If you dont want (or can't due to #nics limitations) to pass a nic per vlan you need a generic trunk port with all applicable vlans, see the following example from a dvSwitch:
1609675203784.png

Inside the VM you then can attach a single nic
1609675345548.png

You can then add VLAN interfaces on top of the single physical/virtual interface (here from a Sophos UTM vm)

1609675443261.png

Hope that helps.
 

Partofthesolution

New Member
Dec 3, 2020
28
7
3
Thanks for reply.

Re topic 2 - it sounds as if the order of NICs your vm sees is not identical to the order you see in the gui.
Im pretty sure that the intial setup with four NICs is as expected. This means:

vmx0 = Network Adapter 1 = WAN portgroup which is connected to "WAN"-Standard vSwitch = only vmnic3 is attached, which is the NIC with connected DSL modem only
vmx1 = Network Adapter 2 = LAN portgroup which is connected to "vSwitch0" Standard vSwitch = only vmnic2 is assigned with connected physical managed switch
vmx2 = Network Adapter 3 = VLAN0101 tagged Portgroup which is connected to "vSwitch0" Standard vSwitch = only vmnic2 is assigned with connected physical managed switch
vmx3 = Network Adapter 4 = VLAN0102 tagged Portgroup which is connected to "vSwitch0" Standard vSwitch = only vmnic2 is assigned with connected physical managed switch

This setup was working until i added the fifth and sixth Network Adapter. I verified this setup by letting clients pull DHCP leases in every VLAN.

After adding more Network Adapters to the VM, this order is shuffled in a unknown way by internal ESXi rules and i dont know why.

Re topic 1
If you dont want (or can't due to #nics limitations) to pass a nic per vlan you need a generic trunk port with all applicable vlans, see the following example from a dvSwitch:
What is a dvSwitch?

I forgot to say, that i only have ESXi free licence. This means, i cant configure other switches than "Standard vSwitch" (Std vSwitch dont have much options).
I of course configured the needed VLANs on my physical attached cisco/linksys switch.
 
Last edited:

Partofthesolution

New Member
Dec 3, 2020
28
7
3
OK guys, the first topic might be a OPNsense/FreeBSD or a ryzen specific topic. I talked to a friend and hes also struggling with this problem with opnsense on a ryzen system (but baremetal).

Topic 2:

Yes, thats the right place to configure VGT (VLAN guest tagging). I did a lot of research in the last days and finaly found the solution last night.

Here are some related VMware KB articles.


Solution:

The portgroup must be assigned to VLAN ID 4095. This is the basic requirement to let your vSwitch passthrough tagged ethernet frames with any VLAN ID from 1 to 4094.

1609691197942.png

Remember: With Standard vSwitch (which is the only available type of vSwitch in ESXi free), you can not filter the allowed VLAN IDs, so in a worst case scenario, an attacker in a guest could change or add the vlan id in the guest and gain access to all available VLANs on the vSwitch.

VLAN IDs can only be filtered with distributed vSwitches, which are available in enteprise licence i think.
 

Rand__

Well-Known Member
Mar 6, 2014
5,381
1,147
113
The portgroup must be assigned to VLAN ID 4095. This is the basic requirement to let your vSwitch passthrough tagged ethernet frames with any VLAN ID from 1 to 4094.
Well that's what it says right there in the screenshot;)

VLAN IDs can only be filtered with distributed vSwitches, which are available in enteprise licence i think.
VMUG ...
But probably overkill for a single host;)

Glad you got it solved
 

Partofthesolution

New Member
Dec 3, 2020
28
7
3
Thanks!

Actually, if you already know what "All (4095)" means, yes, otherwise, its hard to understand that this is the "enable vgt"-switch. :)

Yeah, i thought about VMUG, but i am still struggling with the 200USD/year.... maybe later.
 

Rand__

Well-Known Member
Mar 6, 2014
5,381
1,147
113
Well it says "All" ;)
But its not necessarily like one would expect it to work, I agree.

VMUG usually can be had for 10% less with a code, sometimes even higher rebates are available..
But still enough, certainly; it only makes sense if you run a couple of VMWare products
 

Partofthesolution

New Member
Dec 3, 2020
28
7
3
Is there a newsletter to get informations about discounts?
Whats the best source for news about VMUG promos / discounts?
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
5,381
1,147
113
I am sure you have seen the popup too:
NEW to Advantage: VMware vSphere® 7, VMware vSAN™ 7 and VMware NSX-T™ Data Center! Use code ADVNOW for 10% off your 1 year VMUG Advantage subscription today!
 

bpalob

New Member
Apr 17, 2021
1
0
1
@Partofthesolution: Did you get ESXI7 to run on a APU4? If yes, I´d die to know how you managed to get this done. I got 6.5 at best, and only trouble with USB and Networking somehow... Would wish to go to V7...
 

Partofthesolution

New Member
Dec 3, 2020
28
7
3
Sorry, i never tried to install ESXi on my APU.

The goal was to use a opensense config which works in a ESXi-Guest-VM and my APU.