Configure pfsense with using a single 40gb connection and multiple VLANs?

[Fallen Kell]

I am currently in the process of building a pfsense box (I got a really good deal on a Dell 9020 sff system). This has a 16xPCIe 3.0 expansion slot. I have already sitting around another HP branded dual port mellanox connectx-3 VPI card (and I will probably need to hack some cooling/fan to go over the card). That said, I don't want to use up 2 qsfp+ ports on my switch when my WAN link is currently only 1gbps (possibly getting doubled to 2gbps soon).

I wanted to ask around here to see if anyone believes this is a really bad idea, or if I should try and find qsfp+ to sfp+ converters and/or a qftp+ to sfp+ DAC (mellanox makes one I believe) and connect both ports at 10gbps on the connectx-3. The reason I was trying to avoid that it would essentially defeat the purpose of using the connectx-3 card I have since it would most likely be cheaper to buy a dual 10gbps card for the added price of the cables and/or converter to go from qsfp+ to sfp+. I don't believe there is any kind of bandwidth issue in allowing the traffic to all go over the same physical 40gbps connection with VLAN tagging as the reason (to my understanding) that you typically want 2 ports is so that you can handle the incoming and outgoing data without taxing the ports (so if you want 1gbps routing, you really need 2x1gbps of bandwidth, but in my case, I will have 40gbps). Again, anyone know of any gotchas that I need to look out for or if there is some reason this would not work well?

So my network would essentially be like this (at least this is what I am thinking for eventual rules):
VLANS:
WAN_VLAN (obviously this is used only for the WAN link and the pfsense system)
GUEST_VLAN (routeable to the internet for guests connecting to my wired and wireless)
IOT_VLAN (routeable to the internet for "internet of things" devices that need internet access to properly function)
IOT_ISO_VLAN (not routeable to the internet, but can be accessed from both the PROD_VLAN or MANAGEMENT_VLAN, but can not initiate the connection to those VLANs)
PROD_VLAN (routeable to the internert, and can be connected to from the MANAGEMENT_VLAN)
MANAGEMENT_VLAN (can connect to the PROD_VLAN, will not be able to connect into from internet)

WAN/cable-modem 1gbps ethernet ---> WAN_VLAN tagged port on switch
pfsense box 40gbps qsfp+ dac (configure all VLANs needed in my network, such as WAN_VLAN, GUEST_VLAN, IOT_VLAN, IOT_ISO_VLAN, PROD_VLAN, MANAGEMENT_VLAN) ---> Trunk port with all VLANs on switch
pfsense box 1gbps ethernet --> MANAGEMENT_VLAN tagged port on switch


Thanks.

 

[j_h_o]

I have a similar set up, with a 40Gbps NIC.

What switch are you using? If it's L3-capable, you may wish to have your switch on each of the VLANs, and then only expose 1 VLAN to the pfSense. It is far nicer to have the pfSense only see 2 VLANs (WAN and LAN), and then let your switch handle the routing between the VLANs at wirespeed. You'd bake your ACLs between VLANs into the switch config, instead of the pfSense.

 

[Fallen Kell]

Yes, the switch is L3 (an ICX6610). I will need to look more into creating all the rules for routing on the switch (I know there are quite a few people here who probably have already posted quite a bit of information here about it). I am glad someone else has already effectively done something similar, and will see about just dealing with the WAN <--> LAN routing (pfblockerng, and possibly snort IDS/IPS... this last will depend on load, but I think I have the hardware for it with a i7-4790 CPU and 16GB RAM).

 

[j_h_o]

I think most of the gotchas seem to be making sure you have a DHCP server that can properly handle mulitple scopes, relayed through the Brocade. I have this working with an ubnt EdgeRouter and with various DHCP/DNS servers. I haven't tried it with pfSense hosting DHCP/DNS.

Assuming VLAN 10, 11, 12, and 13 are on various eth 1/1/x ports, and pfSense is on 1/3/1 (expecting untagged vlan 10, at 10.10.0.254), you could do something like:


enable
conf t

vlan 5 name Servers
router-interface ve 5
tag eth 1/3/2
interface ve 5
ip address 10.5.0.1/24

vlan 10 name pfSense
router-interface ve 10
untag eth 1/3/1
interface ve 10
ip address 10.10.0.1/24

vlan 11 name Guest
router-interface ve 11
untag eth 1/1/1
interface ve 11
ip address 10.11.0.1/24
ip helper-address 1 10.5.0.5 #Put your DHCP+DNS server here

vlan 12 name IOT
router-interface ve 12
untag eth 1/1/2
interface ve 12
ip address 10.12.0.1/24
ip helper-address 1 10.5.0.5 #Put your DHCP server here


Then do a static route on the Brocade, so it sends everything it doesn't know, to pfSense:

ip route 0.0.0.0/0 10.10.0.254


And on the pfSense, set up a static route for anything internal.
In the GUI, add a gateway "core":
destination 10.0.0.0/8 all via 10.10.0.1
If you need to bootstrap a fresh install, you can untag a port on the pfSense VLAN, then ssh into the pfSense first, and temporarily add a route via the shell:

route add -net 10.0.0.0/8 10.10.0.1


Once that's working, you can configure ACLs:


ip access-list extended "VLAN11 outbound"
deny tcp any host 10.11.0.1 eq ssh
permit udp any any eq bootps
permit udp any 10.5.0.5 255.255.255.255 eq dns
permit icmp any host 10.5.0.5 echo
permit tcp any 10.0.0.0 0.255.255.255 established
permit ip 10.11.0 0.0.0.255 any
deny ip any 10.0.0.0 0.255.255.255
permit ip any any
interface ve 11
ip access-group "VLAN11 outbound" in


 

[am45931472]

highly interested in this thread. Love pfsense, love my brocade 7250 and 6610. I know that from a performance point of view having the switch do all the inter-vlan routing is the way to go but my networking is not that strong and managing it all from pfsense is much easier, plus nice to have vlan information all centralized in the pfsense gui makes it easy to manage.

 

[j_h_o]

I use pfSense to control internet access from each of the VLANs - I still have control over what devices in each VLAN can do on the internet. Changes here occur, and I make these in the GUI.

The switch ACLs only control what each VLAN can access/do to other VLANs -- and this largely doesn't change: my guest VLAN never needs access to my trusted VLAN, for example - and that basically hasn't changed since I set it up

 

[CorvetteGS]

I have this working with an ubnt EdgeRouter and with various DHCP/DNS servers.

I have been wanting to configure my network this way for a while. (Edgerouter ER-4 and ICX 7150 switch) Do you have IPv6 working on your network? My hold up is I can't figure out how to take the IPv6 PD that my ISP hands to my edgerouter and have something like an IPv6 version of a helper address.