Opnsense and icx6610 confusion

Jason Antes

Member
Feb 28, 2020
128
23
18
Twin Cities
I am trying to set up VLANs for my network. My issue is that only vlan1 and 2 work properly. Pretty sure I have the OPNSense setup correctly for vlans. All vlans except 1 are assigned to 1/3/8 which is ix1 on the firewall. On vlan2 that port is set to dual-mode on the Brocade. Is that why it works when the other vlans on that port don't?

I am also able to ping the vlan gateway address from vlan1 without issue. I cannot ping the VE interfaces on the switch though.

Code:
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree 802-1w
!
vlan 2 by port
tagged ethe 1/3/8
untagged ethe 1/1/48
!
vlan 20 by port
tagged ethe 1/3/8
untagged ethe 1/1/38 to 1/1/47
router-interface ve 20
spanning-tree 802-1w
!
vlan 30 name "IOT Network" by port
tagged ethe 1/3/8
untagged ethe 1/1/36 ethe 1/2/2
router-interface ve 30
spanning-tree 802-1w
!
vlan 40 name DMZ by port
tagged ethe 1/3/8
untagged ethe 1/2/4
router-interface ve 40
spanning-tree 802-1w
Code:
SSH@Thor#sho int brief

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/2      Up      Forward Full 1G    None  No  1    0   cc4e.2416.ac86
1/1/3      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/4      Up      Forward Full 1G    None  No  1    0   cc4e.2416.ac86
1/1/5      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/6      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/7      Up      Forward Full 1G    None  No  1    0   cc4e.2416.ac86
1/1/8      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/9      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/10     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/11     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/12     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/13     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/14     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/15     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/16     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/17     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/18     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/19     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/20     Up      Forward Full 1G    None  No  1    0   cc4e.2416.ac86
1/1/21     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/22     Up      Forward Full 1G    None  No  1    0   cc4e.2416.ac86
1/1/23     Up      Forward Full 1G    None  No  1    0   cc4e.2416.ac86
1/1/24     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/25     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/26     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/27     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/28     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/29     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/30     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/31     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/32     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/33     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/34     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/35     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/36     Up      Forward Full 100M  None  No  30   0   cc4e.2416.ac86
1/1/37     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/1/38     Up      Forward Full 100M  None  No  20   0   cc4e.2416.ac86
1/1/39     Down    None    None None  None  No  20   0   cc4e.2416.ac86
1/1/40     Down    None    None None  None  No  20   0   cc4e.2416.ac86
1/1/41     Up      Forward Full 1G    None  No  20   0   cc4e.2416.ac86
1/1/42     Down    None    None None  None  No  20   0   cc4e.2416.ac86
1/1/43     Up      Forward Full 1G    None  No  20   0   cc4e.2416.ac86
1/1/44     Down    None    None None  None  No  20   0   cc4e.2416.ac86
1/1/45     Down    None    None None  None  No  20   0   cc4e.2416.ac86
1/1/46     Down    None    None None  None  No  20   0   cc4e.2416.ac86
1/1/47     Down    None    None None  None  No  20   0   cc4e.2416.ac86
1/1/48     Up      Forward Full 1G    None  No  2    0   cc4e.2416.acb5
1/2/1      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/2/2      Up      Forward Full 10G   None  No  30   0   cc4e.2416.ac86
1/2/3      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/2/4      Down    None    None None  None  No  40   0   cc4e.2416.ac86
1/2/5      Up      Forward Full 10G   None  No  1    0   cc4e.2416.ac86
1/2/6      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/2/7      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/2/8      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/2/9      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/2/10     Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/3/1      Up      Forward Full 10G   None  No  1    0   cc4e.2416.ac86  Hephaestus-Port
1/3/2      Up      Forward Full 10G   None  No  1    0   cc4e.2416.ac86  Hephaestus-Port
1/3/3      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/3/4      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/3/5      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/3/6      Down    None    None None  None  No  1    0   cc4e.2416.ac86
1/3/7      Up      Forward Full 10G   None  No  1    0   cc4e.2416.ac86  Firewall-Port1_
1/3/8      Up      Forward Full 10G   None  Yes 2    0   cc4e.2416.ac86  Firewall-Port2_
mgmt1      Down    None    None None  None  No  None 0   cc4e.2416.ac86

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2416.ac86
ve20       Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2416.ac86
ve40       Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2416.ac86
ve30       Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2416.ac86
SSH@Thor#sho ip rou
  route                    Show IP routes
SSH@Thor#sho ip route
Total number of IP routes: 4
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
        Destination        Gateway         Port          Cost          Type Uptime
1       192.168.1.0/24     DIRECT          ve 1          0/0           D    1d20h
2       192.168.20.0/24    DIRECT          ve 20         0/0           D    21h3m
3       192.168.30.0/24    DIRECT          ve 30         0/0           D    22h25m
4       192.168.40.0/24    DIRECT          ve 40         0/0           D    49m40s
 
Last edited:

tubs-ffm

Member
Sep 1, 2013
88
26
18
What do you want to achieve?

a) Using the ICX as L2 switch for your VLAN that all are connected via a trunk to OPNsense. OPNsense is the router (router-on-a-stick).
b) Using the ICX as L3 switch and doing routing between OPNsense and ICX and on the ICX?

To me it looks like you mixing these configurations.

I am doing both
For VLAN 20, 30 and 40 the ICX is L2 switching only. In his case no router interface is assigned to the VLAN.
For network "LAN" and "DMZ" the ICX is acting as L3 switch. Here I have assigned router interfaces and routes from adn to OPNsense.

Network.png
 
  • Like
Reactions: Jason Antes

tubs-ffm

Member
Sep 1, 2013
88
26
18
I am new to L3 too. I am only two steps ahead of you and I already got it running.


There is not only one way to go. The question is what you want to achieve with the L3 routing. In my case I am using L2 switching only for VLAN 20, 30 and 40 only because there is no connection to each other. Hosts in these VLANs only can communicate to each other or to the internet. If I need connection between these VLAN I could do by OPNsense as router. But I do not need.

For my networks called DMZ and LAN I do have some limited interconnection. This I also could do with OPNsense as router and firewall rules. But in my case, this was a performance bottleneck I wanted to use the 10G capability of the ICX switch and therefore moved the routing from OPNsense box to the L3 switch. This additionally required to set-up all access rules on the switch. Without the ACL routing is easy, but it would be meaningless to have separate networks. And the ACL only for these two networks created a lot of headache to me.

To use my example above, LAN network 192.168.2.0/24 and DMZ network 192.168.10.0/24 only exit on the L3 switch and each of them is connected to a router interface. The network 192.168.5.0/30 is connecting the switch with router interface ve5 (192.168.5.2) and the OPNsense box with network interface "transport" (192.168.5.1). There is no VLAN between switch and OPNsense. On the switch I need to setup a route for 0.0.0.0/0 to 192.168.5.1 to forward all traffic to the OPNsense box. And on the OPNsense box I need to set-up a gateway on network interface transport to the switch IP 192.168.5.2. And I need to setup routes on OPNsense for the network 192.168.2.0/24 and 192.169.10.0/24 to this gateway.

Maybe not so easy to follow the description. But it will help to search for the right information. The tricky part on OPNsense it to set up this configuration. You need to be connected via any other interface directly to the box to be able to set-up the transport interface, gateway and routes.

On the switch it looks like this: (some cut out)

Code:
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 5 name transport by port
untagged ethe 1/1/3
router-interface ve 5
!
vlan 10 name DMZ by port
tagged ethe 1/1/5 ethe 1/1/9 ethe 1/2/6 ethe 1/2/8
untagged ethe 1/1/11 to 1/1/12 ethe 1/2/5 ethe 1/2/7
router-interface ve 10
!
vlan 20 name IoT by port
tagged ethe 1/1/5 ethe 1/1/7 ethe 1/1/9
!
vlan 30 name Guest by port
tagged ethe 1/1/5 ethe 1/1/7 ethe 1/1/9
untagged ethe 1/1/10
!
vlan 40 name VLAN40 by port
tagged ethe 1/1/9 ethe 1/1/12 ethe 1/2/5
untagged ethe 1/1/1
!
!
!
ip route 0.0.0.0/0 192.168.5.1
ip router-id 192.168.5.2
!
!
!
interface ve 1
ip access-group lan in
ip address 192.168.2.1 255.255.255.0
!
interface ve 5
ip address 192.168.5.2 255.255.255.252
!
interface ve 10
ip access-group dmz in
ip address 192.168.10.1 255.255.255.0
!
!
!
!
 
  • Like
Reactions: Jason Antes

laserpaddy

Active Member
Jul 17, 2017
171
40
28
out there
good explanation-thank you- I integrate unifi and pfsense which was the easiest for a newish person- now I wanted to move some of the access to management proxmox interfaces- etc onto the switch and file transfers etc- seemed like the only way to use a firewall as a firewall only-
My ultimate goal is to firewall ids/ips and routing as seperates- hate having that 1 point of failure PLUS I like the granularity it would give
 

Jason Antes

Member
Feb 28, 2020
128
23
18
Twin Cities
Trying to do the first option. I want to get rid of vlan1 and 2, moving vlan1 to vlan 10 and 2 to 20. 20 will have my ubiquiti unifi AP. Vlan 10 will talk to 20 for wireless printers and the controller for the AP to talk to them. Only the AP controll traffic intiates talks to vlan 10 so I have 2 rules allowing that to a specific IP. Vlans 30 and 40 only initiate talk out to the internet. Vlan 10 will have access to talk to those. The DMZ is for some vm's I am hosting for work access and minecraft stuff.
 

laserpaddy

Active Member
Jul 17, 2017
171
40
28
out there
the only ubiquity products I like are cameras and the xg16- shouldve gotten the edge products but I have the 24lite- too many options- Im trying to get better resolution on the bs on sly
 

Jason Antes

Member
Feb 28, 2020
128
23
18
Twin Cities
Vlan 30 will also need to talk to my AP since that will be my IOT vlan.
Breakdown of vlans:

10 > internal wired network for servers and pc's
20 > internal wireless for phones, printers, etc
30 > IOT
40 > DMZ
 

Jason Antes

Member
Feb 28, 2020
128
23
18
Twin Cities
What do you want to achieve?

a) Using the ICX as L2 switch for your VLAN that all are connected via a trunk to OPNsense. OPNsense is the router (router-on-a-stick).
b) Using the ICX as L3 switch and doing routing between OPNsense and ICX and on the ICX?

To me it looks like you mixing these configurations.

I am doing both
For VLAN 20, 30 and 40 the ICX is L2 switching only. In his case no router interface is assigned to the VLAN.
For network "LAN" and "DMZ" the ICX is acting as L3 switch. Here I have assigned router interfaces and routes from adn to OPNsense.

View attachment 17642
Looking at this, I see that I maybe don't have a "transport" vlan. What's that? Also, is it maybe that I don't have default routes set for the switch?
 

tubs-ffm

Member
Sep 1, 2013
88
26
18
Looking at this, I see that I maybe don't have a "transport" vlan. What's that? Also, is it maybe that I don't have default routes set for the switch?
Look in my visualization some posts above. "Transport" is not an official name. This is how I called the network connection between the two routers: OPNsense and the L3 switch. It is not a VLAN, it is a pure ethernet connection in my case. It only is a VLAN inside the Ruckus. But I also could have assigned the router interface directly to the ethernet port.

I guess you need to get more familiar with L2 switching, L3 routing and routing in general before going to the implementation of a specific scenario. All what you describe above is possible with a pure L2 switching setup and routing on OPNsense box. Generic information to this setup you can find by searching for "router on a stick" in the internet.
 
  • Like
Reactions: Jason Antes

Jason Antes

Member
Feb 28, 2020
128
23
18
Twin Cities
I got it figured out. I had the switch setup ok, it was an issue with OPNSense. I worked into the early morning hammering on this and finally decided to just start from scratch by wiping the firewall config. After that, things started working. Now I just need to get the switch an IP on the ve 10 interface so that I can ssh to it and get the ubiquiti AP's reconfigured. They are passing IP addresses and allowing connections but I can't manage them due to them not getting a new IP (they are DHCP). I tagged the link to the AP's and set the network in the controller, just got to get them to pick up a new IP for management.