Thanks for continuing to upload these, I absolutely wouldn't know about them otherwise.
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Out of curiosity... 
If you guys/girls:
- use a VLAN capable switch (like the ones in this thread or any other)
- and actually use VLANs...
Do you:
- terminate your WAN at the switch and access it via a VLAN?
- or terminate it at your firewall/router?
If you guys/girls:
- use a VLAN capable switch (like the ones in this thread or any other)
- and actually use VLANs...
Do you:
- terminate your WAN at the switch and access it via a VLAN?
- or terminate it at your firewall/router?
I am using my Layer 3 switch as the first Layer 3 device and letting it do the routing between my VLANs.Out of curiosity...
If you guys/girls:
- use a VLAN capable switch (like the ones in this thread or any other)
- and actually use VLANs...
Do you:
- terminate your WAN at the switch and access it via a VLAN?
- or terminate it at your firewall/router?
Clients use the switches VLAN interface as the default gateway.
The switch then has the LAN interface of my Firewall as its default gateway.
Did you ever consider terminating your WAN on the switch (in a VLAN) and using a single pipe to your firewall to do everything? If not, why not? Complexity? Security?
I have considered all the options, but why have a super fast layer 3 switch and let my firewall do the routing, may as well get a layer 2 switch with vlans. (Reason 1)
I also run a VLAN with MTU 9000 so any communication to the other standard MTU 1500 segments needs to be segmented. Again a task I trust to the hardware switch to do faster than my firewall. (Reason 2)
If my firewall goes down for updates, or failure I still have all local connectivity intact. (Reason 3)
There is still a single pipe to the Firewall and keep my setup the same, I can send a trunk port and created a couple of VLAN interfaces on the Firewall but keep the routing on the switch. That is something I thought about doing for fun but have not done just because time and no real gain in my circumstances.
Ugh, sorry, I should have been more clear.
I meant a single pipe to the firewall only for firewall duties, not move layer 3 switching to the firewall.
My point in this was, if you have capable switches, terminating the WAN on the switch (in a VLAN) reduces the need for multi port firewall/NAT devices. This is of course with a caveat that if you're looking at > 1gbps bandwidth, then your firewall/NAT pipe should be sized appropriately (LACP/10g etc).
The flexibility you get from terminating at the switch and doing things like port mirroring (for analysis/IDS rules etc) is an added bonus.
I meant a single pipe to the firewall only for firewall duties, not move layer 3 switching to the firewall.
My point in this was, if you have capable switches, terminating the WAN on the switch (in a VLAN) reduces the need for multi port firewall/NAT devices. This is of course with a caveat that if you're looking at > 1gbps bandwidth, then your firewall/NAT pipe should be sized appropriately (LACP/10g etc).
The flexibility you get from terminating at the switch and doing things like port mirroring (for analysis/IDS rules etc) is an added bonus.
I am only really concerned with LAN<>WAN firewall no matter how I use the Firewall I will get that benefit.
I also run my IPS in line not on a port mirror, this is faster and more powerful than trying to block packets after they have already passed.
PFSense (what I am using) is also an interesting creature in that it does not let you do DHCP for subnets that it does not see as directly connected.
I am using PFSense for DHCP right now, in the next couple of weeks looking to stand up an AD server and run DHCP and possibly DNS from there.
I also run my IPS in line not on a port mirror, this is faster and more powerful than trying to block packets after they have already passed.
PFSense (what I am using) is also an interesting creature in that it does not let you do DHCP for subnets that it does not see as directly connected.
I am using PFSense for DHCP right now, in the next couple of weeks looking to stand up an AD server and run DHCP and possibly DNS from there.
Your setup sounds similar to mine. pfSense has been great to me over the years of experimenting. With the ICX6610 upgrade in my rack from the Netgear switch I had, I decided to get the Brocade to do the L3 routing. I learned ACLs today and I still have some tweaking to do, and I have no idea if I'm doing this right.
For those of you with more networking knowledge, feedback is welcome. Right now, the switch acts as my VLAN gateways. All traffic not routed in the switch goes to pfSense which does LAN-WAN firewall duty, Squid proxy cache, DNS, and DHCP.
pfSense is connected on each VLAN (for DHCP purposes) as 10.1.x.254/24.
Code:
Current configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
!
global-stp
!
!
lag electron-Trunk dynamic id 1
ports ethernet 1/3/1 to 1/3/2
primary-port 1/3/1
lacp-timeout short
deploy
!
lag neutron dynamic id 5
ports ethernet 1/2/2 to 1/2/3
primary-port 1/2/2
lacp-timeout short
deploy
!
lag proton dynamic id 6
ports ethernet 1/2/7 to 1/2/8
primary-port 1/2/7
lacp-timeout short
deploy
!
lag pve1-Trunk dynamic id 4
ports ethernet 1/2/4 to 1/2/5
primary-port 1/2/4
lacp-timeout short
deploy
!
lag pve2-Trunk dynamic id 3
ports ethernet 1/2/9 to 1/2/10
primary-port 1/2/9
lacp-timeout short
deploy
!
lag sw2-Trunk dynamic id 2
ports ethernet 1/3/3 to 1/3/6
primary-port 1/3/3
lacp-timeout short
deploy
!
!
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 2 name VOIP by port
tagged ethe 1/2/4 to 1/2/5 ethe 1/2/9 to 1/2/10 ethe 1/3/1 to 1/3/6
router-interface ve 2
spanning-tree
!
vlan 3 name IOT by port
tagged ethe 1/3/1 to 1/3/6
untagged ethe 1/1/48
router-interface ve 3
spanning-tree
!
vlan 4 name SAN by port
tagged ethe 1/1/37 to 1/1/38 ethe 1/3/1 to 1/3/7
untagged ethe 1/2/2 to 1/2/3 ethe 1/2/7 to 1/2/8
router-interface ve 4
spanning-tree
!
vlan 5 name MGMT by port
tagged ethe 1/2/4 to 1/2/5 ethe 1/2/9 to 1/2/10 ethe 1/3/3 to 1/3/7
untagged ethe 1/1/1 to 1/1/12
router-interface ve 5
!
vlan 6 name APP by port
tagged ethe 1/2/4 to 1/2/5 ethe 1/2/9 to 1/2/10 ethe 1/3/1 to 1/3/6
router-interface ve 6
spanning-tree
!
vlan 10 name TRUSTED by port
tagged ethe 1/1/37 to 1/1/38 ethe 1/3/1 to 1/3/7
router-interface ve 10
spanning-tree
!
vlan 20 name UNTRUSTED by port
tagged ethe 1/2/4 to 1/2/5 ethe 1/2/9 to 1/2/10 ethe 1/3/1 to 1/3/7
router-interface ve 20
spanning-tree
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
jumbo
hostname sw1-br
ip dhcp-client disable
ip dns server-address 10.1.1.1 10.1.1.254
ip route 0.0.0.0/0 10.1.1.254
!
no telnet server
username rich password .....
username admin password .....
snmp-server community ..... ro
snmp-server contact Rich Gannon <rich@richgannon.net>
snmp-server location Core Rack
!
!
clock timezone us Eastern
!
!
ntp
disable serve
server 10.1.1.1
server 10.1.1.254
!
!
web-management session-timeout 3000
!
!
!
!
!
!
!
interface management 1
ip address 192.168.1.1 255.255.255.0
!
interface ethernet 1/1/48
inline power
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 2
ip access-group 102 in
ip address 10.1.2.1 255.255.255.0
!
interface ve 3
ip access-group 103 in
ip address 10.1.3.1 255.255.255.0
!
interface ve 4
ip access-group 104 in
ip address 10.1.4.1 255.255.255.0
!
interface ve 5
ip access-group 105 in
ip address 10.1.1.1 255.255.255.0
!
interface ve 6
ip access-group 106 in
ip address 10.1.6.1 255.255.255.0
!
interface ve 10
ip access-group 110 in
ip address 10.1.10.1 255.255.255.0
!
interface ve 20
ip access-group 120 in
ip address 10.1.20.1 255.255.255.0
!
!
!
access-list 102 permit icmp 10.1.2.0 0.0.0.255 10.1.0.0 0.0.255.255 echo-reply
access-list 102 deny tcp any 10.1.2.0 0.0.0.255 eq ssh
access-list 102 deny tcp any 10.1.2.0 0.0.0.255 eq http
access-list 102 deny tcp any 10.1.2.0 0.0.0.255 eq ssl
access-list 102 deny ip any 10.1.1.0 0.0.0.255
access-list 102 deny ip any 10.1.3.0 0.0.0.255
access-list 102 deny ip any 10.1.4.0 0.0.0.255
access-list 102 deny ip any 10.1.5.0 0.0.0.255
access-list 102 deny ip any 10.1.6.0 0.0.0.255
access-list 102 deny ip any 10.1.10.0 0.0.0.255
access-list 102 deny ip any 10.1.20.0 0.0.0.255
access-list 102 permit ip any any
!
access-list 103 permit icmp 10.1.3.0 0.0.0.255 10.1.0.0 0.0.255.255 echo-reply
access-list 103 permit tcp 10.1.3.0 0.0.0.255 eq http 10.1.10.0 0.0.0.255
access-list 103 deny tcp any 10.1.3.0 0.0.0.255 eq ssh
access-list 103 deny tcp any 10.1.3.0 0.0.0.255 eq http
access-list 103 deny tcp any 10.1.3.0 0.0.0.255 eq ssl
access-list 103 deny ip any 10.1.1.0 0.0.0.255
access-list 103 deny ip any 10.1.2.0 0.0.0.255
access-list 103 deny ip any 10.1.4.0 0.0.0.255
access-list 103 deny ip any 10.1.5.0 0.0.0.255
access-list 103 deny ip any 10.1.6.0 0.0.0.255
access-list 103 deny ip any 10.1.10.0 0.0.0.255
access-list 103 deny ip any 10.1.20.0 0.0.0.255
access-list 103 permit ip any any
!
access-list 104 permit icmp 10.1.4.0 0.0.0.255 10.1.0.0 0.0.255.255 echo-reply
access-list 104 deny tcp any 10.1.4.0 0.0.0.255 eq ssh
access-list 104 deny tcp any 10.1.4.0 0.0.0.255 eq http
access-list 104 deny tcp any 10.1.4.0 0.0.0.255 eq ssl
access-list 104 deny ip any 10.1.1.0 0.0.0.255
access-list 104 deny ip any 10.1.2.0 0.0.0.255
access-list 104 deny ip any 10.1.3.0 0.0.0.255
access-list 104 deny ip any 10.1.5.0 0.0.0.255
access-list 104 deny ip any 10.1.6.0 0.0.0.255
access-list 104 deny ip any 10.1.10.0 0.0.0.255
access-list 104 deny ip any 10.1.20.0 0.0.0.255
access-list 104 permit ip any any
!
access-list 105 permit icmp 10.1.1.0 0.0.0.255 10.1.0.0 0.0.255.255 echo-reply
access-list 105 permit icmp host 10.1.1.254 any echo
access-list 105 permit tcp 10.1.1.0 0.0.0.255 eq http 10.1.10.0 0.0.0.255
access-list 105 permit tcp 10.1.1.0 0.0.0.255 eq ssl 10.1.10.0 0.0.0.255
access-list 105 permit tcp 10.1.1.0 0.0.0.255 eq ssh 10.1.10.0 0.0.0.255
access-list 105 permit tcp 10.1.1.0 0.0.0.255 eq 5900 10.1.10.0 0.0.0.255
access-list 105 permit tcp 10.1.1.0 0.0.0.255 eq asf-rmcp 10.1.10.0 0.0.0.255
access-list 105 permit udp 10.1.1.0 0.0.0.255 eq asf-rmcp 10.1.10.0 0.0.0.255
access-list 105 permit ip host 10.1.1.10 10.1.10.0 0.0.0.255
access-list 105 permit ip host 10.1.1.11 10.1.10.0 0.0.0.255
access-list 105 deny ip any 10.1.2.0 0.0.0.255
access-list 105 deny ip any 10.1.3.0 0.0.0.255
access-list 105 deny ip any 10.1.4.0 0.0.0.255
access-list 105 deny ip any 10.1.5.0 0.0.0.255
access-list 105 deny ip any 10.1.6.0 0.0.0.255
access-list 105 deny ip any 10.1.10.0 0.0.0.255
access-list 105 deny ip any 10.1.20.0 0.0.0.255
access-list 105 permit ip any any
!
access-list 106 permit icmp 10.1.6.0 0.0.0.255 10.1.0.0 0.0.255.255 echo-reply
access-list 106 permit tcp host 10.1.6.39 any eq ssh
access-list 106 permit tcp host 10.1.6.40 any eq ssh
access-list 106 permit tcp host 10.1.6.39 any eq rsh-spx
access-list 106 permit tcp host 10.1.6.40 any eq rsh-spx
access-list 106 permit tcp host 10.1.6.39 any eq asf-rmcp
access-list 106 permit tcp host 10.1.6.40 any eq asf-rmcp
access-list 106 permit udp host 10.1.6.39 any eq asf-rmcp
access-list 106 permit udp host 10.1.6.40 any eq asf-rmcp
access-list 106 permit tcp 10.1.6.0 0.0.0.255 eq ssh 10.1.10.0 0.0.0.255
access-list 106 permit tcp 10.1.6.0 0.0.0.255 eq rsh-spx 10.1.10.0 0.0.0.255
access-list 106 permit tcp 10.1.6.0 0.0.0.255 eq http 10.1.10.0 0.0.0.255
access-list 106 permit tcp 10.1.6.0 0.0.0.255 eq ssl 10.1.10.0 0.0.0.255
access-list 106 deny tcp any 10.1.6.0 0.0.0.255 eq ssh
access-list 106 deny tcp any 10.1.6.0 0.0.0.255 eq http
access-list 106 deny tcp any 10.1.6.0 0.0.0.255 eq ssl
access-list 106 deny ip any 10.1.1.0 0.0.0.255
access-list 106 deny ip any 10.1.2.0 0.0.0.255
access-list 106 deny ip any 10.1.3.0 0.0.0.255
access-list 106 deny ip any 10.1.4.0 0.0.0.255
access-list 106 deny ip any 10.1.5.0 0.0.0.255
access-list 106 deny ip any 10.1.10.0 0.0.0.255
access-list 106 deny ip any 10.1.20.0 0.0.0.255
access-list 106 permit ip any any
!
access-list 110 permit ip any any
!
access-list 120 permit icmp 10.1.20.0 0.0.0.255 10.1.0.0 0.0.255.255 echo-reply
access-list 120 deny tcp any 10.1.20.0 0.0.0.255 eq ssh
access-list 120 deny tcp any 10.1.20.0 0.0.0.255 eq http
access-list 120 deny tcp any 10.1.20.0 0.0.0.255 eq ssl
access-list 120 deny ip any 10.1.1.0 0.0.0.255
access-list 120 deny ip any 10.1.2.0 0.0.0.255
access-list 120 deny ip any 10.1.3.0 0.0.0.255
access-list 120 deny ip any 10.1.4.0 0.0.0.255
access-list 120 deny ip any 10.1.5.0 0.0.0.255
access-list 120 deny ip any 10.1.6.0 0.0.0.255
access-list 120 deny ip any 10.1.10.0 0.0.0.255
access-list 120 permit ip any any
!
!
!
!
!
!
!
endIf you have your pfSense connected to each VLAN...how is your switch doing layer 3 routing?? Where's the gateway for each VLAN? on the switch or pfSense?
Edit: I think I see it now...your pfSense IP on each VLAN is x.x.x.254/24, right?
My current network design is using vlans to segregate different types of devices, having my 6450 handle all the internal routing duties (with trunks to other switches which only act as layer 2 w/ vlan devices). The 6450 will forward everything to the firewall (probably pfSense, maybe Sophos UTM - IPV6 Prefix-Delegation limits being the deciding factor). The 6450 handles DHCP for IPV4 devices and hands out ULA addresses for IPV6 for internal use. I'm using ACLs on between vlans where I want to restrict traffic (e.g. IoT stuff, security cameras). I've desktop tested all this configuration and it works as expected, as soon as I get some armored fibre cables I'm waiting on I'll be installing this to replace my currently flag /24 network.
Distributing the v6 Prefix-Delegation is the only unsolved problem I have, I may end up building a box running WIDE-DHCP to pull over PDs from my firewall device and distribute those routable subnets to devices in their respective vlans. since the 6450 can't do that. If I can't get that to work I'll need to put an interface of the firewall in the vlans I want to have routable IPV6.
Overall the design goal is to have the network operate independent of whether I have an operational internet connection or whether the firewall is available at the moment. I had a previous iteration where the firewall handled all the routing and of course it deterred me from experimenting with different firewall options or upgrading the firewall since it meant the whole network was dead in the water.
Distributing the v6 Prefix-Delegation is the only unsolved problem I have, I may end up building a box running WIDE-DHCP to pull over PDs from my firewall device and distribute those routable subnets to devices in their respective vlans. since the 6450 can't do that. If I can't get that to work I'll need to put an interface of the firewall in the vlans I want to have routable IPV6.
Overall the design goal is to have the network operate independent of whether I have an operational internet connection or whether the firewall is available at the moment. I had a previous iteration where the firewall handled all the routing and of course it deterred me from experimenting with different firewall options or upgrading the firewall since it meant the whole network was dead in the water.
I'm moving to a router on a stick setup slowly over here. My plan is to ultimately terminate the WAN at one of my 6450s and run a pfSense HA cluster on a couple of T730 thin clients with Mellanox CX-3 cards. I use pfSense to firewall/route between about 6 vlans and the WAN for visibility into FW events and because it's convenient to stand up a RADIUS server in 5 minutes with the freeradius package. When I get more lab infrastructure up and running I'm planning to have my switches route storage traffic so that won't pass through pfSense.Out of curiosity...
If you guys/girls:
- use a VLAN capable switch (like the ones in this thread or any other)
- and actually use VLANs...
Do you:
- terminate your WAN at the switch and access it via a VLAN?
- or terminate it at your firewall/router?
Last edited:
I see in your video that you ended up with the 6.3 CFM fans. How have the temps been in your 48P so far? Have the fans spun up to full speed yet?
Was just about to order the 10.8 CFM Sunon fans based on your video until I realized how quiet the 6.3 CFM ones were!
Temps are good, I let it run a solid hour before I got the temps posted in the video. However as I warned that was not under heavy load.
They have never gone to stage 2 and spun up to the faster speed. Personally I would prefer the 10.x CFM fans as it does not need to be silent just quiet and the 6.x CFM fans are silent. I know I still have the stage 2 speeds should thermals go up but just piece of mind I suppose.
I have a 24P on the way right now, if it is also loud like the 48P was I will get the proper fans and see how they turn out. I also got some nice shelves cheap on ebay to hold the switches since none of them had ears. It's cheaper than buying ears so if they work out I might just do a quick follow up video featuring the shelves and the 10CFM fans.
Is it possible to pipe the web interface on these over an SSH tunnel? I've been wracking my brain this morning trying to come up with some kind of Amazing™ (read: janky) solution to the web UI only being available over HTTP.
The best solution I've thought of so far is using an otherwise completely empty vlan to connect directly to one of my docker host pis running traefik and proxying it from there, does that sound viable?
The best solution I've thought of so far is using an otherwise completely empty vlan to connect directly to one of my docker host pis running traefik and proxying it from there, does that sound viable?
If you are specifically not happy with HTTP because its not secure but HTTPS is viable, there are steps to create the certificates and configuration to use HTTPS.
I think this video covers it:
I think this video covers it:
Wow, now I feel dumb. For some reason I had it stuck in my head that these just didn't do https. Thanks, that just saved me from cobbling something hilariously awful together.
Re-did my home network based on what I was thinking... and it actually turned out pretty good. I even removed the standalone DHCP/DNS server and got pfSense to play nice with a layer 3 switch and doing DHCP/DNS without compromising on the routing speed between VLANs.Out of curiosity...
If you guys/girls:
- use a VLAN capable switch (like the ones in this thread or any other)
- and actually use VLANs...
Do you:
- terminate your WAN at the switch and access it via a VLAN?
- or terminate it at your firewall/router?
Also terminated my WAN directly on the switch (on a VLAN, no SVI), so no there is a single 10gb pipe to pfSense.
The "performance" of the WAN seems to have improved slightly, although it could just be placebo. (I have symmetric gigabit at home)
Mind adding more detail so I can understand what you did? I might do the same?
I figure you created vlan interfaces on PFSense so that you can do DHCP but use the switches vlan interfaces for routing. As for the connection to PFSense from the switch, a trunk port?
A simple network diagram would do wonders as well as any relevant switch configuration.
Edit: Oh I think I just sort of realized one thing you were saying you wanted to do that made no sense to me. Instead of your connection from the wall going directly into the WAN interface of your PFSense box, you have it going into the switch on its own vlan so it can talk to anybody other than the PFSense WAN interface, so when you kept saying having your WAN directly to the switch that is what you meant? I don't actually see any reason to do that and it kind of gives me the shudders from a security standpoint but I think I finally understand it now.
I'll draw a diagram in a bit (don't have my Visio machine handy, it's at the office).
The problem I was trying to solve (in a somewhat elegant manner) was multi-fold:
- pfSense won't do DNS/DHCP for anything that's not an interface in it. But, if we do that with a layer 3 switch, the question becomes, which device is going to be the gateway for the VLAN? The switch or pfSense?
- Preserve line rate performance between VLANs, at the switch.
- Why do we need to have anything more than a single NIC in a firewall/pfSense device *IF* we're using a layer 3 switch?? The L3 switch IS a router, all we need to do is pass traffic back and forth between the firewall, depending on the rules you setup. If you want to setup rules between VLANs, do it at the switch (it IS a router after all), if you want to setup rules between your internal network and the WAN, that's when you go to your firewall.
- How do I do packet analysis/capture, traffic analysis, IDS etc etc on the WAN, without limiting myself to the hardware/software that runs the firewall?
So, in trying to solve these issues, the lightbulb moment actually came from someone here...
@Blue)(Fusion - you're the man! (or woman)Let's take a layer 3 switch and create 3 VLANS on it.
1. The WAN VLAN - This is just a VLAN, add untagged ports to it (at least two, so you can mirror your WAN port and do any additional stuff with that traffic). There is no IP address on this VLAN, no SVI, nothing. Make sure you add the TRANSIT port as tagged to this VLAN.
On the pfSense side, create a VLAN that has the matching ID on the same Nic that is used for the TRANSIT pipe. Don't add an interface based on this VLAN, go into interface assignment, and for the WAN, just choose it from the drop down.
Nothing special needs to be done beyond this, other than when we're finally done, reboot the firewall, and do release/renew on the WAN (assuming it's DHCP based).
2. The transit VLAN - This is the VLAN that connects the switch to the firewall. In my case, all it has is a single 10g port, dual-mode (the native VLAN is the transit VLAN), but there's more than one way to skin this cat. The IP address for this VLAN on the switch side is 172.16.0.1/30 (in my case, we only need two IP addresses).
On the pfSense side, there is *no* VLAN for this (remember we added an untagged port on the switch side). The IP address on the pfSense side is static IPv4, 172.16.0.2/30. I named this interface TRANSIT in pfSense. It does not require any additional gateway or anything like that.
Go to the firewall rules and allow ICMP on this interface.
This will allow you to ping the switch IP (172.16.0.1) from pfSense and the interface IP on the pfSense side (172.16.0.2) from the switch console. These two guys are happy as clams at this point.
On your switch add a default route (0.0.0.0/0) to point to pfSense (172.16.0.2). With that the switch will send all unknown traffic on the transit VLAN to pfSense. If you try to do a nslookup or anything like that on the switch console at this point, it won't work, because we haven't set up the rules yet.
3. Your home/whatever VLAN where the (some of the) devices are - Add whatever untagged ports you need on the switch but also add the transit port as tagged. This will ensure that any unknown traffic from this VLAN will be tagged by the time it hits pfSense over the transit pipe.
In my case, on the switch side, I added a few untagged ports to this, added the tagged transit port and gave it an IP of 10.10.10.1/24. That IP address is important...
On the pfSense side, I created a VLAN with the same ID on the same NIC that the transit interface runs on. Then I added an interface based on that VLAN, and gave it a static IP v4 address of 10.10.10.254/24. No gateway or anything needed. But, we're not done yet.
Enable the pfSense DHCP server on this interface, and give it the following settings.
The important parts are:
- Make sure the DHCP range does not include .1 or .254, as we're using those IP addresses, but other than that, nothing special.
- The gateway is set to the IP of the switch for this VLAN. That way, when a device connects, gets an IP address from DHCP, they get a gateway that is on the switch itself. If they access anything else on the switch itself, all L3 routing will be at the switch. The switch will only send traffic to the TRANSIT pipe if it does not know how to route it.
But we're not done yet...
Your device gets an IP address, DHCP is working, the DNS server (you'll be getting the x.x.x.254 IP as the DNS server) is ... inaccessible. That's because we haven't set up the firewall rules yet. So... for this interface in pfSense:
I added the ICMP rule for testing, it's not required. Essentially allow DNS traffic as tagged traffic over the TRANSIT pipe.
Now if you do a nslookup on the device, it'll work and resolve the IP, but still no Internet... We're not done yet.
Allow the interface net traffic on the TRANSIT interface to whatever you choose, right now it's set to anything in my case. When you're setting up the rule in pfSense, the drop down for source will have the "xxxxinterface net" (which is essentially the IP range). And the switch knows how to route this traffic.
Assuming we've done everything right, you should be up and running with Internet access on that device VLAN. From this point on, knock yourself out adding more VLANS and/or firewall rules.
p.s. (I hope I got everything down accurately. This is actually good, as it's documenting it for me as well.
Last edited:
Did you just assume I'm hooman?So, in trying to solve these issues, the lightbulb moment actually came from someone here...
Really nice write up! I know what I'm going to be spending my Saturday doing. Just when I think I've figured it out......destroy it all and build it better!
Just for clarification. Do you have device hostnames registered in DNS through DHCP somehow with this setup? So far that's the only thing keeping me from ditching pfSense almost entirely. Although I'm slowly just adding A records in my local DNS and using more and more static IPs. Your setup still looks like what I'm aiming for.