Running pfSense virtualized but serving the Proxmox as well

[cromo]

I am configuring my t730 now with a 4xI340-T4 PCIe to run proxmox with multiple VMs, including pfsense which will be the firewall for my LAN and proxmox+VMs themselves.

I want to:

1. Pass-through the Intel PCIe to pfSense and have it handle firewall, NAT, DHCP and other services.
2. Have my Ubiquity AC-Pro connected to one of the I340 ports, together with few more devices, like Apple TV and my personal desktop.
3. Have my VMs route through pfSense as well.
4. Have my Proxmox also accessible from the same LAN handled by pfSense.

Now, this is all fairly obvious, except for the last bit. I would set Proxmox to a static IP address and exclude it from the pfSense's range, which is a sane thing to do, but that still doesn't help if the pfSense VM doesn't get up for some reason. How do I access it then, if all my physical devices are connected to I340s which are passed-through but not served ? I am now debating whether to connect the AC-Pro or my personal Desktop's Ethernet to the Realtek port and have it served to pfSense as a bridge, but that's less than optimal. I was also wondering if and how could I go about with SR-IOV solving it, where Proxmox would provide an in-place LAN routing if pfSense VM were down.

I am fairly versed in Linux net administration, just that I always did it the "traditional", non-virtualized way, so any help would be appreciated.

 

[zack$]

Seems like a chicken and egg situation.

I don't know the answer to no. 4 but how can the pfsense VM give the hypervisor an ip if the hypervisor is required to boot the VM?

 

[cromo]

Seems like a chicken and egg situation.

I don't know the answer to no. 4 but how can the pfsense VM give the hypervisor an ip if the hypervisor is required to boot the VM?

VM does not give the hypervisor an IP in my scenario, the IP would be configured statically since that's a good practice. But even then obtaining an IP from a VM via DHCP is perfectly doable.

 

[msg7086]

4xI340-T4

4 ports or 4x4 ports?

If you are using only 4 ports, than an always-online switch shall solve the chicken egg problem (when pfsense is non-functional, you still have your LAN interconnections).

If you are using 16 ports and solely rely on pfsense for LAN connections, then it will be hard.

-- Yes, I think you'll need a cable towards Realtek eth port.

 

[mbosma]

I've been running pfsense on proxmox for years now.
Instead of passing through a network device I opted for bridging the connection.
Create a seperate bridge for wan to connect to isp or datacenter feed.
Enabling vlan awareness on your LAN side bridge allows you to add vlans within pfsense without adding virtual interfaces.

If however you want to use pci-e passthrough to use tcp-offloading and qos you'll have to either bridge the pci-e card to the motherboard interface via a cable or add a virtual interface.

 

[Cyclone]

I haven't even been able to get that far. I have a very similar setup and similar desires. I also have a HP T730 and Intel I340 quad-port NIC. For the most part, all seems well, but I cannot get PCI Passthrough to work properly for the life of me.

I updated Grub with "amd_iommu=on", updated the modules listing, did an "update-grub", then an "update-initramfs -u -k all", and rebooted. I can see the remapping messages from AMD-Vi, see my NIC in a IOMMU group alongside its PCI-E controller, and attached the card to the pfSense VM with "hostpci0: 01:00,pcie=1". I can see in the node's syslog that its enabled for the VM and I can see using qm monitor's "info pci" that it is "passed" to the VM. However, the pfSense ISO installer says that no NICs are found.

I tried playing with various options to no avail. At the moment, I've removed the PCI entry on the VM and created Linux Bridges for two of the ports to be able to connect and utilize pfSense successfully. Any thoughts on what I may be missing to get PCI passthrough working vs. using bridges?

Cromo,
In my setup, I have one port on the NIC going to my Arris modem's built-in switch, one port going to my managed switch, and two ports unused currently. Meanwhile, I have the built-in Realtek port connected to the same switch and set up with a static IP for the same subnet as pfSense. My AP will eventually take up one of the other ports on the Intel NIC, but it is currently connected to the Arris. Stuff like my IoT devices are currently connected through the pfSense and thus, being double-NATed for now while I get everything working just right. I haven't done your #3 yet, but that's mostly because I've been working to get this VM running right before I spin up any others. I intend for the other VMs to bridge to the Realtek port unless I find that too unstable.

 

[Cyclone]

For anyone coming along later, I finally figured out the missing piece of the puzzle in the above. I was searching Reddit and came across this post, so I figured it was worth a shot. I turned on passthrough again using "hostpci0: 01:00" without any pcie statement. Fired up my pfSense VM and immediately all worked just peachy. It say all 4 ports of the NIC (since I passed along "all functions") and I was able to configure it quickly. I can't tell you how many hours I spent trying to get this working to find such a simple "solution". I hope this helps others!