Xcp-ng + Opnsense VM + ATT WAN bypass issue

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

marcoi

Well-Known Member
Apr 6, 2013
1,577
308
83
Gotha Florida
I'm transition off vmware software due to vmug changes. I selected XCP-NG for my main lab servers. I am having issues with one vm which is my Opnsense VM with ATT Ont bypass enabled. I got everything except the ATT wpa working. Not sure what the issue is. In VMware, I have external interface assigned to Distributed Group Port and have it using Vlan 0. Then I assign that virtual network to wan nic of Opnsense VM. Im using [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass - Page 2 WPA method described in the link to setup opnsense.

I know a few out there are doing Opnsense with ATT Ont bypass, anyone running that setup under xcp-ng? If so what are you doing to get the wpa to work?
I dont know if the issue with the xcp nic driver, not changable mac address or vlan 0 on eth pif is not working.

Any help would be greatly appreciated. This is the last VM i need to move over to get my main lab cutover.

DPG policies on vmware
1738041834295.png


WPA_cli status on working vmware, I removed some info. TCPDump shows data going to att on vmx0 interface.
------------------------------
Selected interface 'vmx0'
bssid=01:80:c2:00:00:03
freq=0
ssid=
id=0
mode=station
pairwise_cipher=NONE
group_cipher=NONE
key_mgmt=IEEE 802.1X (no WPA)
wpa_state=COMPLETED
ip_address= (removed but ATT IP)
address=(removed but the mac from gateway certs)
Supplicant PAE state=AUTHENTICATED
suppPortStatus=Authorized
EAP state=SUCCESS
uuid= (removed)
 

marcoi

Well-Known Member
Apr 6, 2013
1,577
308
83
Gotha Florida
Anyone know if vlan 0 works on xcp?

im seeing weird traffic when i try using vlan0 vif

00:02:17.749705 (removed) (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from (removed) (oui Unknown), length 300, xid 0xe2a57e2b, secs 3, Flags [none]
Client-Ethernet-Address (removed) (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Requested-IP (50), length 4: 192.168.100.244
Client-ID (61), length 7: ether (removed)
Hostname (12), length 8: "OPNsense"
Parameter-Request (55), length 10:
Subnet-Mask (1), BR (28), Time-Zone (2), Classless-Static-Route (121)
Default-Gateway (3), Domain-Name (15), Domain-Name-Server (6), Hostname (12)
Unknown (119), MTU (26)

I removed the mac address and the yellow line is what is added when i try vlan0 vif.

Next thing to try is a new vm with just base setup and wpa setup for att lan. Just incase the migration from vmware broke something in opnsense.
 

marcoi

Well-Known Member
Apr 6, 2013
1,577
308
83
Gotha Florida
some more testing looks like the vlan 0 tag on vif is not really working. I tested working system on vmware by setting up a smart switch and vlan the ports so it stips the vlan tag info per the pfatt guide. but when i did that, i notice the same tcp dump info as non working opnsense on xcp.

As soon as i removed the smart switch and directly plugged in the cable, the vmware opnsense tcp dump show connecting and traffic.

So i either have to confirm xcp-ng does indeed do vlan0 like vmware dist switch can or go back to using netgraph of pfatt
 
Last edited:

zachj

Active Member
Apr 17, 2019
224
134
43
Why aren’t you doing pass through of the bare metal nic to the opnsense vm?

I wouldn’t personally but a software switch between the firewall and the internet; that’s a lot of trust to put in the software.
 

marcoi

Well-Known Member
Apr 6, 2013
1,577
308
83
Gotha Florida
Why aren’t you doing pass through of the bare metal nic to the opnsense vm?

I wouldn’t personally but a software switch between the firewall and the internet; that’s a lot of trust to put in the software.
I need to be able to move the vm around when i do maintenance on a system, with pass through I cant do migrations/

The vmware dist switch with vlan 0 worked correctly. ATT ont requires vlan 0 traffic with the bypass gateway method I am using.
I dont think xcp can do proper vlan 0 which is why i am seeing an issue. I am going to try netgraph method next and also ask on xcp forum which I just got access to post on.

lets see which works first if any lol
 

zachj

Active Member
Apr 17, 2019
224
134
43
I need to be able to move the vm around when i do maintenance on a system, with pass through I cant do migrations/

The vmware dist switch with vlan 0 worked correctly. ATT ont requires vlan 0 traffic with the bypass gateway method I am using.
I dont think xcp can do proper vlan 0 which is why i am seeing an issue. I am going to try netgraph method next and also ask on xcp forum which I just got access to post on.

lets see which works first if any lol
with dynamic direct path I/O you can vmotion a vm having a pass through device as long as there’s an identical device in the target host.
 

marcoi

Well-Known Member
Apr 6, 2013
1,577
308
83
Gotha Florida
with dynamic direct path I/O you can vmotion a vm having a pass through device as long as there’s an identical device in the target host.
im coming off vmware and going onto xcp-ng. I dont think xcp has that feature for dynamic direct path.