pfSense Same NIC port has DHCP if configured as LAN but no DHCP if configured as VLAN on same NIC port

[pwm80211]

I have been trying to setup some VLANs in a pfSense2.7 ProxMox VM with a 4 port Intel NIC card. I cannot get DHCP to work with the VLAN configured port but DHCP works fine if it's just configured as a regular second LAN port.

Any help would be greatly appreciated.

 

[louie1961]

Going to need a lot more info to properly assist you. How are you making the NIC available to pfSense? Did you do a passthrough? Did you assign each port to a separate VMBR in pfSense? What hardware is assigned to which port(s) in pfSense, i.e., WAN, LAN, OPT1, etc. ?

 

[pwm80211]

I assigned all 4 NIC ports to vmbr1 through vmbr4 on the ProxMox Host.

vmbr1 (NIC physical port 0)
vmbr2 (NIC physical port 1)
vmbr3 (NIC physical port 2)
vmbr4 (NIC physical port 3)

For testing I have a ubuntu24.04 VM with net0 assigned to vmbr3

NIC physical port 2 is connected to NIC physical port 3

In the pfSense VM I have:

net0 assigned to vmbr1
net1 assigned to vmbr2
net2 assigned to vmbr3
net4 assigned to vmbr4

WAN assigned to vtnet0
LAN assigned to vtnet1
LAN2 assigned to vtnet3
VLAN20 assigned to VLAN20 on vtnet3
VPNK assigned to tun_wg0

WAN port is connected to ISP
LAN port is connected to switch
LAN2 port is connected to vmbr3

WAN IPv4 Configuration is DHCP, DHCP 6
LAN IPv4 Configuration is Static IPv4 with an address of 192.168.15.1/24
LAN2 IPv4 Configuration is Static IPv4 with an address 192.168.10.1/24
VLAN20 Ipv4 Configuration is Static IPv4 with an address 192.168.20.1/24

VPNK is a WireGuard site to site tunnel and is functioning properly.

VLAN20 interface is disabled.

DHCP servers are enabled for interfaces LAN (192.168.15.100 to 200) and LAN2 (192.168.10.100 to 200)

The WAN, LAN, LAN2 and VPNK function properly with 100+ network devices.

The ubuntu test VM gets DHCP address 192.168.10.100 assigned.

I connect the vmbr3 (NIC physical port 2) to the switch and restart systemd-networkd on ubuntu test VM. The ubuntu test VM gets DHCP address 192.168.15.182 assigned.

I reconnect the vmbr3 (NIC physical port 2) to vmbr4 (NIC physical port 3) and restart systemd-networkd on ubuntu test VM. The ubuntu test VM gets DHCP address 192.168.10.100 assigned.

I disable LAN2 and enable VLAN20
I enable DHCP server for VLAN20 (192.168.20.100 to 200)

I restart systemd-networkd on ubuntu test VM.

The ubuntu test VM does not get a DHCP address assigned.

 

[TrevorH]

Did you typo DHCP as DNS in your first post?

 

[pwm80211]

Yes, I corrected it

 

[louie1961]

A couple of thigs I don't understand:

First off, how is Proxmox connected to the network? How do you reach the Proxmox web interface? Is there a second NIC you are using for that?

Second, what does the rest of your network look like? WAN is what? Do you have a switch of some kind? Is it a managed switch? How is it connected to everything else?

Third, you said "For testing I have a ubuntu24.04 VM with net0 assigned to vmbr3? What exactly does this mean?

Fourth you said "NIC physical port 2 is connected to NIC physical port 3" Why on earth would you do this?

The way I do this on my Proxmox setup is as follows:

My Proxmox box physically has four NICs: Two Realtek NICs on the motherboard and a dual port SFP+ Intel X520 NIC. I use the two motherboard NICs for my redundant WAN connections (Xfinity cable modem and T-Mobile 5G). The first port on my X520 NIC is the LAN port on pfsense and is also the connection to the Proxmox interface. The second port on my X520 NIC is used only to connect directly to Proxmox if pfSense is down. Inside of Proxmox, I have 3 VMBRs set up, one for each WAN NIC and one for the LAN. In my pfSense VM I have three virtual NICs, one on VMBR0, one on VMBR1, and one on VMBR2.

VMs on my Proxmox box that hosts pfSense can connect to proxmox over VMBR0. VMBR0 also connects to my managed switch, and my other proxmox nodes and/or things like myNAS also connect to the managed switch.

Here's what my /etc/network/interfaces file looks like in Proxmox. This is really where everything special is configured. There's nothing special inside of pfSense. I have my vlans defined there, but it would look the same if it was runnig bare metal actually.

auto lo
iface lo inet loopback

# ===== LAN Trunk (X520 Port 0) =====
auto enp3s0f0
iface enp3s0f0 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp3s0f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4092

# Primary Proxmox management on VLAN 100 (tagged on vmbr0)
auto vmbr0.100
iface vmbr0.100 inet static
        address 192.168.10.6/24
        gateway 192.168.10.1

# Proxmox host on VLAN 3 (Storage VLAN, non-routed)
auto vmbr0.3
iface vmbr0.3 inet static
        address 192.168.3.10/24

# ===== Emergency / Rescue Management (X520 Port 1) =====
# Dedicated bridge on the second port with a private subnet and NO gateway.
# Plug your laptop or an isolated switch into enp3s0f1 and set your laptop to 192.168.99.10/24, for example.
auto enp3s0f1
iface enp3s0f1 inet manual

auto vmbr3
iface vmbr3 inet static
        address 192.168.99.6/24
        bridge-ports enp3s0f1
        bridge-stp off
        bridge-fd 0
        # no 'gateway' here by design


# ===== WAN1 (Realtek 2.5g) =====
auto enp1s0
iface enp1s0 inet manual

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0

# ===== WAN2 (Realtek 1g) =====
auto enp2s0f0
iface enp2s0f0 inet manual

auto vmbr2
iface vmbr2 inet manual
        bridge-ports enp2s0f0
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

In Proxmox VLAN 100 is my management interface on my dedicated management VLAN, and VLAN 3 is dedicated, non-routed storage vlan that connects directly to my NAS

 

[pwm80211]

@louie1961 Thanks for your response.

Here's more connection detail.

I am using the pfSense LAN port to connect to the ProxMox web interface. I use the MB NIC to connect to ProxMox if pfSense is down.

The pfSense WAN connection (VMBR1 port 0) is connected to the ISP ONT. The pfSense LAN port (VMBR2 port 1) is connected to a ICX6450. The ICX6450 ports are all on the default vlan1 for now.

I setup the Ubuntu VM with the dedicated NIC port (port 2) just for testing. My other ProxMox VMs are using VMBR2 for their network connections. The VLAN is using (VMBR4 port 3). I was connecting port 2 to port 3 just to test DHCP for VLAN20. I hope that makes a bit more sense.

I am new to VLANs and realize there are countless combinations using ProxMox, pfSense and the ICX6450 router firmware. I thought it might easier to start with pfSense VLANs first but I'm obviously missing something with setting up a pfSense DHCP server with a VLAN. I'll take a look at ProxMox VLANs and what you are doing with your environment.

 

[louie1961]

OK, so my first advice is to not use your pfSense as a switch. It does a very poor job of that. Generally speaking, the only things that should connect directly to pfSense are your WAN(s) and your LAN. LAN needs to connect to the switch via an untagged/trunk port. With this configuration the network port can communicate all VLANs/VLAN traffic to the switch, and the switch can sort out which devices get which traffic. In your switch, you will generally have the option of configuring each port as tagged to a single VLAN (and you can decide which ports connect to which VLAN by the tag assigned) or you can configure each port as a trunk port that passes all the VLANs. But to connect to a trunk port the device on the other end needs to be VLAN aware and able to handle multiple VLANs. A VLAN aware setup in Proxmox, TrueNAS and other Linux systems can do that. Certain wireless access points can do that, as can managed switches. So in my switch, I have access ports tagged to my home VLAN for my Ring alarm, one for my Synology NAS, etc. I also have tagged access ports for my non-routed, dedicated storage VLAN. I have assign all of my storage devices and proxmox nodes static IPs on that VLAN, since the traffic never goes through pfSense. This keeps my storage network very fast. Then I have untagged/trunk ports for all of my Proxmox nodes and my wireless access pooint, which is VLAN aware. I am ignoring hybrid ports for the moment for the sake of simplicity.

Now, before you say I am contradicting myself, a VMBR in Proxmox (or linux in general) is nothing more than a virtual switch. So inside the Proxmox node that runs my pfSense, Proxmox VMs communicate with pfSense over the VMBR. They are not directly attached to any of pfSense ports.

 

[pwm80211]

I did some messing around with the ICX6450 and setup a couple of tagged and untagged VLAN ports. I tried getting an address through those ports with a laptop and the VM but I could not get an IP with either machine. I then set a static IP for the laptop within the VLAN address space (192.168.20.2) and ran nmap and saw the pfSense base address 192.168.20.1 with ports 53, 80 and 443 open.

The VLAN packets are making it from the ICX6450 to pfSense.

I could log into the pfSense GUI from the VLAN. I then set a WAP management VLAN to the same VLAN with a static IP and it was visible and the web interface was accessible. I then set a static IP for the VM and I could ping everything in the VLAN except the base address 192.168.20.1.

I then added a pass all firewall rule for the VLAN. I still cannot ping the VLAN base address or anything outside (LAN or WAN) of the VLAN. From the regular LAN I can ping 192.168.20.1 but nothing in the VLAN.

This learning curve is frustrating.

 

[louie1961]

I hear you on the frustration. Inside of pfSense, just to go through the checklist with you, you should have your vlans tagged to a parent interface. I tag min to the LAN interface, they don't each get tagged to something different








Then for every VLAN you set up you should also set up a DHCP server

 

[louie1961]

Then just for good measure under the DNS resolver general settings I add all my DHCP leases and reservations to the DNS resolver

 

[pwm80211]

Other than the DNS resolver settings, that is how I have it configured but no DHCP is happening in the VLAN.

I setup a mirror port on the ICX6450 and monitored the VLAN port for the ubuntu VM. I restarted systemd-networkd on the VM but didn't see any DHCP traffic. I do see a broadcast DHCP Discover message periodically.

572    206.354089    0.0.0.0    255.255.255.255    DHCP    340    DHCP Discover - Transaction ID 0xeb4e1b73

 

[pwm80211]

The WAN, LAN and VPN tunnel have been working very reliably. I have not seen anything in either the ProxMox or pfSense logs but I'm starting to wonder if there is something messed up with my pfSense VM.

@louie1961 Based on what you've shared and what I've seen online, this shouldn't be so difficult to setup.

 

[louie1961]

Are you using Kea DHCP or the older ISC DHCP? My understanding is that Kea DHCP still lacks some features as implemented in pfSense. I am using the older version. I tried the new version last year and had significant issues with it not working. I am waiting for the ISC version to be fully deprecated in pfSense before I make the move.

 

[pwm80211]

I am using ISC. It's been very reliable for me except for this VLAN "experience".

Your question got me thinking differently about this. The ICX6450 is allowing traffic from the port VLAN and pfSense serves the pfSense web interface yet I can't ping anything outside the VLAN. What kind of rules or routes do you have in pfSense for the VLANs? I could see port 53 and 443 on 192.168.20.1, but I didn't see 67 or 68 being open.

 

[TrevorH]

So that packet you see outbound is a broadcast message sent to everything connected to its ethernet port. It basically says "HELLO out there, is there a DHCP server and can I have an ip address please". If it's not getting a response that means either it's never actually making it out on the wire or the response is not making its way back. You will need to allow UDP ports 67/68 inbound on any firewall on that is on the DHCP requester so it can receive the replies. You'd have to do a similar trace on the DHCP server to see if that broadcast packet ever arrives there and if it sends a reply back.

 

[pwm80211]

I don't recall seeing anyone mentioning rules for ICMP, DHCP etc. when they were setting up the pfSense VLANs but it seems like it's necessary in my case.

 

[sic0048]

Is the problem with your Proxmox network setup? Louie1961's example is a great snapshot into what you need.

First, it is important to realize that while it is easy to add a vlan tag on the default bridge adapter being used in a VM or LXC container, you cannot simply add a vlan tag on the default bridge adapter for the root PVE device. This is why Louie has created a VLAN on the host in addition to the default bridge connection. In his example the default bridge connection is vmbr0, but he sending his traffic over VLAN 100 through the use of the vmbr0.100. As seen here:

# Primary Proxmox management on VLAN 100 (tagged on vmbr0)
auto vmbr0.100
iface vmbr0.100 inet static
address 192.168.10.6/24
gateway 192.168.10.1

If you don't add a VLAN connection for the host, all of the host's traffic is simply going out the default bridge connection without a VLAN tag associate with it. This sounds like what you are experiencing.

Check out this video as it does a good job walking through all of this networking setup on the Proxmox side.

 

[louie1961]

Here's the rules for my guest network, which is really the most locked down



The NET_RFC1918 alias stands for the 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16 networks. Management ports are 22, 80, and 443 Firewall rules apply from the top down. So first is IP addresses blocked by pfBlockerNG-devel, then I block access to management ports on the firewall, then I allow all other ports to the firewall. I block DNS over HTTPS and DNS over TLS as my connection from the DNS server is TLS encrypted already and I force my client devices to port 53 basically. I have one special rule for my daughter to access my wife's website. Then I block all other internal networks and finally traffic can flow to the WAN

 

[pwm80211]

@louie1961 @sic0048 Thank you very much. I was just thinking about how I could get WireShark to sniff my ProxMox network. I would/might have eventually saw there were no tags.

The VLAN learning curve has been challenging. I have an existing pfSense site to site VPN tunnel that's been working great for a few years and I'm sure I will mess something up adding a VLAN layer underneath it.