pfSense and Brocade - Some routing and layer 3 questions (need help with moving Layer 3 to switch)

[MrSliff]

Hey together,

was lucky to get an ICX6450 for cheap and already switched from my old switch to the new one. Everything is working with the old setup, but i want to change some things related to routing VLANS in my Network.

What is set up right now:
- pfSense Firewall/Router with different VLANs and routing set up on pfSense; the VLANs have all their own interface addresses (i.e. VLAN10 -> 192.168.10.1)
- ICX6450 set up with same VLANs connected via LAG to pfSense
- Set up the VLANs which should be routed directly on the switch with their router interfaces and ip addresses (so i.e. for VLAN10 -> ve10 and IP 192.168.10.254 and so on)

The routes on the switch were set up automatically:

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          192.168.1.1     ve 2          1/1           S    2h8m
2       192.168.1.0/24     DIRECT          ve 2          0/0           D    2h8m
3       192.168.10.0/24    DIRECT          ve 11         0/0           D    12h32m
4       192.168.20.0/24    DIRECT          ve 20         0/0           D    12h32m
5       192.168.30.0/24    DIRECT          ve 30         0/0           D    12h32m
6       192.168.40.0/24    DIRECT          ve 40         0/0           D    12h31m
7       192.168.50.0/24    DIRECT          ve 50         0/0           D    12h31m

At the moment my pfSense is doing all the work, so inter-VLAN traffic on the switch will go out to pfsense and will be routed back to the switch (as far as i could see with traceroute). So traffic from VLAN 30 to VLAN 20 will go to Gateway 192.168.30.1 (on pfSense) and then to VLAN 20. This is as expected, since i set up DHCP to push the VLAN-Interfaces on pfSense as the standard Gateway to the clients. Also the clients with Static IPs have set the standard gateways as 192.168.xxx.1 on pfSense.


So i would like to achieve, that the traffic between VLANs 2,10,20,30,40,50 will be routed directly on the switch and VLAN100 will be kept on PFSense, since this is the Guest VLAN for which i want to restrict access to my Network. At least VLANS 20 and 30 should be routed on the switch, because i want to upgrade my Servers and my Computers to 10GBit. If i could keep the setup on pfSense it would be nice, but not a must.

Maybe somebody could guide me a little bit through this or give me help how to start from here.

 

[kpfleming]

Glad to help; what you want to do sounds fairly straightforward.

Step 1: which version of the firmware do you have on the 6450, and are you using the "S" (layer 2) or "R" (layer 3) flavor?

 

[MrSliff]

Glad to help; what you want to do sounds fairly straightforward.

Step 1: which version of the firmware do you have on the 6450, and are you using the "S" (layer 2) or "R" (layer 3) flavor?

Im using the R version:

UNIT 1: compiled on Apr 23 2020 at 10:57:06 labeled as ICX64R08030u
                (9871112 bytes) from Primary ICX64R08030u.bin
        SW: Version 08.0.30uT313

 

[kpfleming]

OK, you're on the latest software release and have layer 3 functionality, that's good.

You'll want to start by making 'router interfaces' in each of the VLANs for which you want the 6450 to handle routing; these interfaces will be given addresses in the subnets on those VLANs, so that the 6450 can be used as a 'gateway' in those VLANs. In the 08 series software, that means using the 'router-interface' command to create an interface in each target VLAN, and then using the 'interface ve' command to configure the interfaces with addresses.

If you can get all of those interfaces created, and are able to ping all of the interface addresses from other nodes on your network, you'll have the first step completed.

 

[MrSliff]

vlan 2 name Mgmt by port
tagged ethe 1/1/1 to 1/1/4
untagged ethe 1/1/23
router-interface ve 2
!
vlan 11 name "Management Network" by port
tagged ethe 1/1/1 to 1/1/5 ethe 1/1/48
router-interface ve 11
!
vlan 20 name "Server Network" by port
tagged ethe 1/1/1 to 1/1/4 ethe 1/1/48
untagged ethe 1/1/9 to 1/1/12
router-interface ve 20
!
vlan 30 name "Wired Clients" by port
tagged ethe 1/1/1 to 1/1/4
untagged ethe 1/1/13 to 1/1/18 ethe 1/1/47
router-interface ve 30
!
vlan 40 name "Wireless Clients" by port
tagged ethe 1/1/1 to 1/1/5 ethe 1/1/48
untagged ethe 1/1/37
router-interface ve 40
!
vlan 50 name "Smarthome Network" by port
tagged ethe 1/1/1 to 1/1/5 ethe 1/1/48
untagged ethe 1/1/45 to 1/1/46
router-interface ve 50
!
vlan 100 name "Guest Network" by port
tagged ethe 1/1/1 to 1/1/5 ethe 1/1/48

interface ve 2
ip address 192.168.1.5 255.255.255.0
ip address 192.168.1.254 255.255.255.0 secondary
!
interface ve 11
ip address 192.168.10.254 255.255.255.0
!
interface ve 20
ip address 192.168.20.254 255.255.255.0
!
interface ve 30
ip address 192.168.30.254 255.255.255.0
!
interface ve 40
ip address 192.168.40.254 255.255.255.0
!
interface ve 50
ip address 192.168.50.254 255.255.255.0

Ping from VLAN 40 to VLAN 20 interface:

ping 192.168.20.254

Ping wird ausgeführt für 192.168.20.254 mit 32 Bytes Daten:
Antwort von 192.168.20.254: Bytes=32 Zeit=6ms TTL=64
Antwort von 192.168.20.254: Bytes=32 Zeit=18ms TTL=64
Antwort von 192.168.20.254: Bytes=32 Zeit=4ms TTL=64
Antwort von 192.168.20.254: Bytes=32 Zeit=1ms TTL=64

Ping-Statistik für 192.168.20.254:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 1ms, Maximum = 18ms, Mittelwert = 7ms

 

[LodeRunner]

Here's a post I wrote up for another user. It's on a 7150-C12P running 8.0.95 but I think the relevant syntax is the same. I'm using a Windows server for DHCP. Your config above looks good, unless you also need DHCP from a single server, then you'll need the helper-address statement. And don't forget the reverse routes on the pfSense side.

Spoiler

ICX7150-C12 Router#sh run
Current configuration:
!
ver 08.0.95eT213
!
stack unit 1
  module 1 icx7150-c12-poe-port-management-module
  module 2 icx7150-2-copper-port-2g-module
  module 3 icx7150-2-sfp-plus-port-20g-module
  stack-port 1/3/1
  stack-port 1/3/2
!
global-stp
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 11 by port
tagged ethe 1/1/1
untagged ethe 1/1/11
router-interface ve 11
!                                                               
vlan 12 by port
tagged ethe 1/1/1
untagged ethe 1/1/3
router-interface ve 12
!
vlan 20 by port
tagged ethe 1/1/1
router-interface ve 20
!
ip dhcp-client disable
ip route 0.0.0.0/0 172.16.21.2
!

!                                                               
interface ve 11
ip address 10.100.11.1 255.255.255.0
!
interface ve 12
ip address 10.100.12.1 255.255.255.0
ip helper-address 1 10.100.11.2
!
interface ve 20
ip address 172.16.21.1 255.255.255.0
end

Switch IP and route tables:

ICX7150-C12 Router#sh ip add
        IP Address       Type      Lease Time       Interface
       10.100.11.1       Static    N/A             ve11
       10.100.12.1       Static    N/A             ve12
       172.16.21.1       Static    N/A             ve20
ICX7150-C12 Router#sh ip rout
Total number of IP routes: 4
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
STATIC Codes - v:Inter-VRF
        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          172.16.21.2     ve 20         1/1           S    18m57s
2       10.100.11.0/24     DIRECT          ve 11         0/0           D    26m26s
3       10.100.12.0/24     DIRECT          ve 12         0/0           D    28m5s
4       172.16.21.0/24     DIRECT          ve 20         0/0           D    19m28s
ICX7150-C12 Router#

pfSense gateway and routes:
View attachment 21045
View attachment 21046

Windows VM in VLAN 11, IP 10.100.11.2/24, running DHCP server, showing client with IP 10.100.12.2/24:
View attachment 21047

 

[kpfleming]

You will also need a static route in the ICX to tell it how it can reach the VLAN 100 network (the next-hop will be the pfSense machine, probably at 192.168.10.1 unless you want to send that traffic over some other VLAN).

 

[MrSliff]

Ok, thanks very much for your suggestions and help!

It seems a bit complicated in my setup to get this done. Since my pfSesne is already doing the routing between the VLANs and also acts as a DHCP server for the clients i think i may try a different approach. I also have docker containers with static ips which dont have the ability to be set up with custom routes.

So i going another way. Since i only have 2 Clients yet which can do 10GBit and pfSense is completely able to route 1gbit easily i just set up two gateway in the switch and set up custom routes between my PC and the unraid Server, so i can copy files with 10GBit. The rest will go to my pfSense and from there to the specific VLANs.

Nevertheless I will have work on it in the future when i will get a faster internet connection. 1GBit Internet is fine for now and maybe 2.5GBit internet could be handled by my pfSense, so no need to change my running system for now.

This is the custom route directly through the switch to my unraid server:

  1    <1 ms    <1 ms    <1 ms  192.168.30.254
  2    <1 ms    <1 ms    <1 ms  xxxxxxx.unraid.net [192.168.20.3]

This is the default route to another server in my server VLAN:

  1    <1 ms    <1 ms    <1 ms  192.168.30.1
  2    <1 ms    <1 ms    <1 ms  192.168.20.21

But thank you anyways!

 

[kpfleming]

That's a reasonable configuration for now, certainly. Glad you got it working.