Can you use pfSense as a VM and link it to a vSwitch with no Physical NIC?

[Steve-o]

Hello all, I am not an expert by any means I just like to play around in my home lab. I just bought a new server for Christmas, but I been beating my head on the table for a couple of days now trying to get this working and I don't think I can do what I want to do. Long story short I have 3 ESXi host's and each one has a RJ-45 10GB dual card in it. and I am trying to link them all together as well as a TrueNAS VM without a 10GB switch. The 10GB link will only be for ESXi stuff. FYI, I do have vCenter running as a VM.

What I thought I could do is setup pfSense VM on my main server (ESXi-03) that will also running TrueNAS then passthrough the dual NIC card to pfSense as well as a vNIC then make a vSwitch with no physical NIC and link the vNIC to it. The Dual NIC card would be connected to other 2 ESXi servers (ESXi-01 & 02). Then in pfSense make a bridge between all 3 NICs. After doing this my hopes were short lived. Only ESXi-01 & 02 could see each other and as well as pfSense but I could not see ESXi-03 or the TrueNAS VM. But the TrueNAS VM and ESXi-03 could see each other but nothing else. It seems like the vSwitch is not really working as a switch maybe it's not doing broadcast to the vNIC in pfSense. See attached diagram on how my network is laid out and what I was trying to do. I know I should just by a switch but since I only have 10GB RJ-45 card the RJ-45 switches are pretty pricey for me right now. If I knew better, I would have got all SFP+ cards because though switches are much cheaper.

If you guys have any suggestions, please let me know.

Thanks

If you guys were curious this is my setup
ESXi-01:
PowerEdge T630
2x CPU E5-2620 v3
96GB Ram
1TB M.2 970 EVO
4x 4TB SAS HGST running RAID-5

ESXi-02:
PowerEdge T430
2x CPU E5-2640 v3
128GB Ram
500GB 870 Evo
4TB Seagate Skyhawk

ESXi-03:
Supermicro 230700-2U-X10DRH-CT-16BL-S3-2PS
2x CPU E5-2690 V3
64GB Ram
1TB M.2 970 EVO
8x 8TB SAS HGST

 

[zer0sum]

Should work just fine...sounds more like a firewall policy issue.

Start with just getting TrueNAS communicating with pfsense, and then move to getting it communicating through pfsense to another host

 

[Steve-o]

Should work just fine...sounds more like a firewall policy issue.

Start with just getting TrueNAS communicating with pfsense, and then move to getting it communicating through pfsense to another host

Hey zer0sum thanks for the reply,

Yeah that was one of my thoughts as well was a firewall problem but when I created the bridge I set the firewall rules to allow any. I even disable the firewall and it still didn't help. Only way that I could get TrueNAS to see pfSense was to configure The virtual NIC to have a IP in pfSense. But since I have a bridge I cannot set that virtual NIC to the same IP range due to the DHCP of the bridge. I even tried to disabling DHCP on the bridge and enabling the virtual NIC DHCP. But that only allowed TrueNAS to see pfSense. And other servers still could see each other but could not see pfSense or TrueNAS.

 

[zer0sum]

Hey zer0sum thanks for the reply,

Yeah that was one of my thoughts as well was a firewall problem but when I created the bridge I set the firewall rules to allow any. I even disable the firewall and it still didn't help. Only way that I could get TrueNAS to see pfSense was to configure The virtual NIC to have a IP in pfSense. But since I have a bridge I cannot set that virtual NIC to the same IP range due to the DHCP of the bridge. I even tried to disabling DHCP on the bridge and enabling the virtual NIC DHCP. But that only allowed TrueNAS to see pfSense. And other servers still could see each other but could not see pfSense or TrueNAS.

Might be an obvious question, but are you allowing promiscuous mode for the virtual interfaces that pfsense is using?

 

[Steve-o]

Might be an obvious question, but are you allowing promiscuous mode for the virtual interfaces that pfsense is using?

Can you enable that on each VM vnic or is it only on the vSwitch? I will have to look into that when I get home tonight. Once again thank you

 

[zer0sum]

Can you enable that on each VM vnic or is it only on the vSwitch? I will have to look into that when I get home tonight. Once again thank you

It's set at the virtual switch or port group level

 

[Steve-o]

@zer0sum you are the man!!! that worked they you