Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[StarsAndBars]

I had my two ICX 6610 stacked using the 40Gb ports with MPO fiber. About 50 meters (more than 150 feet) so this should be easy.

And yes, if you follow fohdeesha's detailed directions, you can easily use the first two ports for stacking at 40Gb/s each and then break out the other two ports for 8 extra 10Gb/s ports. The only limitation on using the break out ports is you will be using an MPO -> 4xLC break out cable, so you can't connect to 10Base-T cards, only SFP+ "stuff" that can accept an LC optical transceiver.

Awesome. Can you please give me more details/link to the gear you used to stack them? Such as the modules you slot into the 40gb ports and the MPO fiber cable you used (pre-terminated, I assume?) I already am using the 4 x breakout cables today and they work great.

Thank you very much!

 

[clcorbin]

Awesome. Can you please give me more details/link to the gear you used to stack them? Such as the modules you slot into the 40gb ports and the MPO fiber cable you used (pre-terminated, I assume?) I already am using the 4 x breakout cables today and they work great.

Thank you very much!

The ICX 6610 is very forgiving on transceivers. Pretty much any QSPF28 module with MPO connector should work. When I bought mine, I pretty much bought the cheapest ones I could find on ebay and they have worked perfectly.

These have worked in my ICX 6610s:

• Brocade 57-1000267-01 XBR-000232 4 x 16GB QSFP (works just fine at 40Gb)
• Brocade 57-1000129-01 40GBASE-SR4 QSFP+ 40GbE 100m
• XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS (These work, but are LC, so can't work with break out cables, as least none that I am familiar with)

 

[dasbooter]

bro you put igmp in the top search box, not in the member box lmao

View attachment 42601

¯\_(ツ)_/¯. In my defense the search bar blacks out when you open the options and no longer looks like a box.

Thanks so much fodheesha. Oh well there goes any modicum of respect I had left on the Internet.

PS I'm not your bro, bro ;⁠-⁠). Lol thanks for helping me get also these sfp+ ports licensed.... Now if I could only figure out what they are for

 

[SeRiusMe]

https://forums.servethehome.com/index.php?threads/layer-3-switch-w-pfsense.23236/post-216621

Thank you Kapone for pointing me in the right direction. I've been studying the method you posted so I'm completely sure that I understand it before doing or replying anything about it. I plan on applying your "TRANSIT" configuration. I'll bring the 6610 to my desk and raise an opnsense device for doing tests. I've seen that the new Kea dhcp server can allocate segments out of the VLAN interfaces so I may check it. But I'm also concerned about the DNS resolver, as I run Adguard on the same machine, pointing to the native opnsense unbound running on another port.
We'll see.

 

[SeRiusMe]

While I study migrating to L3 at the 6610, I received a 6450 that I want to be my office switch. Currently, in the basement homelab, I have a Tplink running L2 only with some Vlans defined. Opnsense machine that does the routing (10.1.2.251 @Trusted VLAN 2) and the ISP link is directly connected to it.
I plan on use the 6610 as a routing switch, replacing the tp-link, and the 6450 connected to it as a simple L2 switch.

While I prepare my final strike, I'm configuring the 6450. I've setup a trunk "uplink" port to the tplink all vlans tagged on both, two untagged ports for my computer and a printer, and then a dual-mode trunk port for the wlan Ap.

But It doesn't seem to work right. If the PC is connected through wire, it does get dhcp address but can't communicate to other lan segments, including internet. While if I connect it via that wifi Ap, it seems to have internet and inter-vlan communication, but it doesn't get a dhcp address.

What I'm missing in the 6450 configuration?
In a pure L2 configuration, does that VLAN 2 (Trusted) needs the `router-interface` and the ve?
I only need a management Ip for the switch on the VLAN 3, but now I'm afraid of kicking me out of the switch. Is there a way to set the switches Ip without this?

Spoiler: sh run

Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
module 1 icx6450-24p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
no legacy-inline-power
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 name LAN by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/2/1 to 1/2/4
untagged ethe 1/1/3 ethe 1/1/24
router-interface ve 2
!
vlan 3 name SYS by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/2/1 to 1/2/4
!
vlan 4 name WRX by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/2/1 to 1/2/4
!
vlan 5 name IOT by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/2/1 to 1/2/4
!
vlan 6 name CTV by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/2/1 to 1/2/4
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname ICX6450
ip dhcp-client disable
ip dns server-address 10.1.2.251
ip route 0.0.0.0/0 10.1.2.251
!
no telnet server
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
disable serve
server 10.1.2.251
!
!
!
!
!
interface ethernet 1/1/1
port-name UPLINK
!
interface ethernet 1/1/2
port-name AP Front
dual-mode 3
inline power
!
interface ethernet 1/1/3
port-name PC Redcomet
!
interface ethernet 1/1/24
port-name PR Laser
!
interface ve 2
ip address 10.1.2.3 255.255.255.0
!
!
!
!
!
!
!
!
!
end

Spoiler: sh vlan

Total PORT-VLAN entries: 6
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1)   4   5   6   7   8   9  10  11  12  13  14  15
Untagged Ports: (U1/M1)  16  17  18  19  20  21  22  23
   Tagged Ports: None
   Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 2, Name LAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1)   3  24
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2   3   4
   Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 3, Name SYS, Priority level0, Spanning tree Off     
Untagged Ports: None
   Tagged Ports: (U1/M1)   1
   Tagged Ports: (U1/M2)   1   2   3   4
   Uplink Ports: None
DualMode Ports: (U1/M1)   2
Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 4, Name WRX, Priority level0, Spanning tree Off
Untagged Ports: None
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2   3   4
   Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 5, Name IOT, Priority level0, Spanning tree Off
Untagged Ports: None
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2   3   4
   Uplink Ports: None
DualMode Ports: None                                         
Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 6, Name CTV, Priority level0, Spanning tree Off
Untagged Ports: None
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2   3   4
   Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled

Spoiler: sh int brief

sh int brief

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Up      Forward Full 1G    None  Yes N/A  0   cc4e.2452.4140  UPLINK     
1/1/2      Up      Forward Full 1G    None  Yes 3    0   cc4e.2452.4140  AP Front   
1/1/3      Up      Forward Full 1G    None  No  2    0   cc4e.2452.4140  PC Redcomet
1/1/4      Down    None    None None  None  No  1    0   cc4e.2452.4143             
1/1/5      Down    None    None None  None  No  1    0   cc4e.2452.4144             
1/1/6      Down    None    None None  None  No  1    0   cc4e.2452.4145             
1/1/7      Down    None    None None  None  No  1    0   cc4e.2452.4146             
1/1/8      Down    None    None None  None  No  1    0   cc4e.2452.4147             
1/1/9      Down    None    None None  None  No  1    0   cc4e.2452.4148             
1/1/10     Down    None    None None  None  No  1    0   cc4e.2452.4149             
1/1/11     Down    None    None None  None  No  1    0   cc4e.2452.414a             
1/1/12     Down    None    None None  None  No  1    0   cc4e.2452.414b             
1/1/13     Down    None    None None  None  No  1    0   cc4e.2452.414c             
1/1/14     Down    None    None None  None  No  1    0   cc4e.2452.414d             
1/1/15     Down    None    None None  None  No  1    0   cc4e.2452.414e             
1/1/16     Down    None    None None  None  No  1    0   cc4e.2452.414f             
1/1/17     Down    None    None None  None  No  1    0   cc4e.2452.4150             
1/1/18     Down    None    None None  None  No  1    0   cc4e.2452.4151             
1/1/19     Down    None    None None  None  No  1    0   cc4e.2452.4152             
1/1/20     Down    None    None None  None  No  1    0   cc4e.2452.4153             
1/1/21     Down    None    None None  None  No  1    0   cc4e.2452.4154             
1/1/22     Down    None    None None  None  No  1    0   cc4e.2452.4155             
1/1/23     Down    None    None None  None  No  1    0   cc4e.2452.4156             
1/1/24     Down    None    None None  None  No  2    0   cc4e.2452.4140  PR Laser   
1/2/1      Down    None    None None  None  Yes N/A  0   cc4e.2452.4140             
1/2/2      Down    None    None None  None  Yes N/A  0   cc4e.2452.4140             
1/2/3      Down    None    None None  None  Yes N/A  0   cc4e.2452.4140             
1/2/4      Down    None    None None  None  Yes N/A  0   cc4e.2452.4140             
mgmt1      Down    None    None None  None  No  None 0   cc4e.2452.4140             

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve2        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2452.4140             
ve3        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2452.4140             
ve4        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2452.4140             
ve5        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2452.4140             
ve6        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2452.4140

 

[kapone]

But I'm also concerned about the DNS resolver, as I run Adguard on the same machine, pointing to the native opnsense unbound running on another port

https://forums.servethehome.com/index.php?threads/adguard-a-good-reality-check.46276/

 

[SeRiusMe]

https://forums.servethehome.com/index.php?threads/adguard-a-good-reality-check.46276/

Thanks Kapone. What a coincidence, I also own a Tesla. I suppose you realized that most of the telemetry filtered by adguard is the communication between the Tesla and their Api mainly to serve the phone app. I had to whitelist a server to bring back the car status while at home. Perhaps that was the reason you had almost 50% requests blocked.

I don't know if you have Adguard installed independently, but there's plugin for Opnsense. I have installed like that.
I have the Opnsenses unbound configured as upstream DNS server so I can overwrite internal requests to services I proxy via haproxy. It was a pita to arrange, and I'm very confident that the slightest change will make that stop working.
I also thought about the DHCP server in Adguard, and how it could replace Opnsenses one if doesn't work on a switch L3 config, but as it is installed by the plugin and it configures the interfaces, I don't know if it could work.

Any hint on the above issue I'm having with the 6450? Since then I've tried looping the management port into an untagged Vlan3 port and making it get an ip by dhcp. I can ssh into it, but as soon as I remove the vlan 2 ve and router interface, it stops talking to my computer. I know is still there because I can ping it from the opnsense. I don't know what is going on. I re-reading all, looking for a pure L2 configuration to see where I went wrong.

 

[dasbooter]

Does anybody know if it is possible and/or useful to use a virtual lab like EVE-NG and simulate the 6 series and 7 series brocade switches?

 

[fohdeesha]

Does anybody know if it is possible and/or useful to use a virtual lab like EVE-NG and simulate the 6 series and 7 series brocade switches?

no

 

[kapone]

I suppose you realized that most of the telemetry filtered by adguard is the communication between the Tesla and their Api mainly to serve the phone app. I had to whitelist a server to bring back the car status while at home. Perhaps that was the reason you had almost 50% requests blocked.

Already knew that. I barely use the App, and Tesla sends way more telemetry to the mothership (which will go anyway, once it's off my wifi) than the App needs. Eh.

I don't know if you have Adguard installed independently, but there's plugin for Opnsense.

I'm using the OPNsense plugin.

I have the Opnsenses unbound configured as upstream DNS server so I can overwrite internal requests to services I proxy via haproxy.

Got it. I suspect that may still work with some other upstream like Quad9, because my internal DNS server (Windows AD) is set up as the first upstream, but only for my own domain and I can fudge with DNS entries there.

I also thought about the DHCP server in Adguard, and how it could replace Opnsenses one if doesn't work on a switch L3 config, but as it is installed by the plugin and it configures the interfaces, I don't know if it could work.

Never tried it and don't intend to. I need a Windows domain...

Any hint on the above issue I'm having with the 6450?

I looked at it briefly, but I probably don't understand what you're trying to do. For e.g.

I have a Tplink running L2 only with some Vlans defined. Opnsense machine that does the routing (10.1.2.251 @Trusted VLAN 2)

If your Tplink is L2 only and has VLANs defined..then I don't understand this reference to "OpnSense does the routing @VLAN 2". Your Opnsense should have all VLANS defined as interfaces on it? (That'd be the L3 part...)

And...

If the PC is connected through wire, it does get dhcp address but can't communicate to other lan segments, including internet.

I don't quite understand this. I'm assuming you mean the PC is hardwired to the 6450 on 1/1/3 (based on your port naming) and that port belongs to VLAN 2 and there's no VLAN "awareness" on the PC end. Who's serving the DHCP here? Opensense? That'd imply VLAN 2 is defined as an interface in it?

I've setup a trunk "uplink" port to the tplink all vlans tagged on both, two untagged ports for my computer and a printer, and then a dual-mode trunk port for the wlan Ap.

So, the Tplink and the 6450 are connected on 1/1/1 which has no untagged traffic on it, right? And then VLAN 2 on the 6450 passes tagged traffic on 1/1/1 and has a VE on it with an IP address of 10.1.2.3/24. I don't see any config from the Tplink so I'm assuming its configured correctly and working.

If all of the above is true, then:
- You should be able to ping your Opnsense on 10.1.2.251 from the 6450 console and vice versa.
- If 10.1.2.3 is the only IP address on the 6450, you should be able to ping it from some other VLAN (other than VLAN 2) as long as Opnsense is configured to pass traffic correctly...

I think what you're running into is a combination of untagged/tagged traffic and incorrect firewall rules in Opnsense. Remember Opnsense firewall rules apply on incoming traffic on any interface. Also, you may know this already, but when you move to a TRANSIT type configuration, the firewall rules become slightly trickier because all traffic comes on a single interface and there's no VLANs in OPNsense. You use "networks/CIDR ranges" as the "source".

 

[kapone]

I've seen that the new Kea dhcp server can allocate segments out of the VLAN interfaces so I may check it.

I have not played with Kea yet, but it seems to have rough edges at the moment.

 

[SeRiusMe]

First, thank you Kapone for your help. This is something clearly escapes to my limited network knowledge. I've been some days struggling already, and since I still haven't found a solution.

Never tried it and don't intend to. I need a Windows domain...

Fair enough. There has been so many years since I had to work with a Windows domain controller. I think it was a r2003.

If your Tplink is L2 only and has VLANs defined..then I don't understand this reference to "OpnSense does the routing @VLAN 2". Your Opnsense should have all VLANS defined as interfaces on it? (That'd be the L3 part...)

I'm so sorry, I was trying to clarify the Ip shown on the configuration attached. The Opnsense is configured as a router-on-stick, hardwired via 10gbe cabling to a trunk port on the tplink. This connection has all the VLANs tagged, and the opnsense has all defined and has an Ip on all, A la 10.1.X.251. It is the DNS, as described and the DHCP using ISC. In NAT I only have defined two open ports, as I use the services of one of those companies that uses a deployed service to forward ports. Don't pay attention to it.

So, the Tplink and the 6450 are connected on 1/1/1 which has no untagged traffic on it, right?

Right. Plain RJ45 1gb with all VLANs tagged on both sides, for the moment.

And then VLAN 2 on the 6450 passes tagged traffic on 1/1/1 and has a VE on it with an IP address of 10.1.2.3/24.

Yes. Since then, I moved the VE to VE3 on the Management VLAN 3... So now it is 10.1.3.3. But doesn't make a difference. I tried to add the 6450 management port to the system and it didn't went well. activated dhcp client for it. Gave the 10.1.3.3 ip on the dhcp and looped to the 1/1/2 port configured as untagged 3. (Moved the the AP to port 3). I wanted to remove VEs for all interfaces thinking it would become a pure L2, but as soon as I removed that last VE I could not establish connection to the switch although this Management ip. The switch continued to work, as I could ping it from the switch.

I don't see any config from the Tplink so I'm assuming its configured correctly and working.

Sorry, here it is. Nothing fancy. All things like STP IGMP Snooping are disabled. It has the management Ip on VLAN 2. I'll be moving it to 3.

Spoiler: TpLink sh run

T1700G-28TQ#show run
!T1700G-28TQRev3


#
vlan 2
name "LAN"
#
vlan 3
name "SYS"
#
vlan 4
name "WRX"
#
vlan 5
name "IOT"
#
vlan 6
name "NVR"
#
#
switch 1 provision T1700G-28TQRev3
ip igmp snooping version v2
spanning-tree mode rstp
no ip routing
user name ****** privilege admin secret ****************************
ip ssh server                
system-time ntp UTC+01:00 10.1.3.251 150.214.94.5 12
ip http secure-session timeout 30
ip http session timeout 30
#
#
#
interface vlan 2
ip address 10.1.2.5 255.255.255.0
description "LAN"
no ipv6 enable

....

# HERE's THE 6450 CONNECTED
interface gigabitEthernet 1/0/5
eee
switchport general allowed vlan 2-6 tagged
no switchport general allowed vlan 1
switchport pvid 3            
spanning-tree

# Other ports....
....

# HERE's THE OPNSENSE CONNECTED
interface ten-gigabitEthernet 1/0/28
switchport general allowed vlan 2-6 tagged
no switchport general allowed vlan 1
switchport pvid 3
spanning-tree

#
#

I think what you're running into is a combination of untagged/tagged traffic and incorrect firewall rules in Opnsense. Remember Opnsense firewall rules apply on incoming traffic on any interface. Also, you may know this already, but when you move to a TRANSIT type configuration, the firewall rules become slightly trickier because all traffic comes on a single interface and there's no VLANs in OPNsense. You use "networks/CIDR ranges" as the "source".

This is interesting and strange, because ATM I only have FW rules for the VLAN 5 that is IOT. Preventing from reaching any other VLAN but with exceptions so I can manage it, and some hosts can access NAS for backups and such.
I've been looking at the FW when connected via RJ45 and I can't see any blocked hits.

I don't understand what you say about untagged/tagged traffic. Do you mean that the untagged port the PC is connected, is not correctly tagging the frames? Or that the response doesn't arrive back with the correct tag?
...is it possible to dual-mode this port 3 and 3? I guess not. So if that's the problem.... would it be coming form the tplink?
Seems that if the traffic has to go from the 6450 edge to the router and come back, it doesn't. Strangely, when connected to the AP, it seems to work. That should discard a problem with the firewall.

Today I reverted to dynamic Ip and my computer got an address while yesterday didn't.
So now the problem seems to be that wired connections doesn't reach any other network segment, including internet. I can ping the router in the Trusted VLAN, and access other servers on that VLAN but nothing else.

Routing isn't happening? Wired comms doesn't get to the opnsense?
I don't have defined any routes on the opnsense, and I can't see anything wrong on my PC routes or ARP table.

What should be the "gateway" on the 6450? Should be the OpnSense, or the Tplink? I mean the default route.
Should I have ip-helpers defined? I guess not, and L2 should be enough for DHCP.
Should I have VEs in all the VLANs? And then an address in each interface? None of them for a pure L2?
You see the 6450 configuration okay?

If only I could see a basic configuration for a L3 switch + pure L2 switch downstream, with two and 3 sample VLANs... I've been tracking for this on the forum since two days without success. I still don't get that no-one has posted a sample. I'm convinced that the problem may be the language breach and I'm not doing the right searches.

EDIT: I almost forgot! Pings when my PC is connected via Wifi are all OK, a mess when hardwired.
When I ping from my PC and I get the "Unreachable network" it shows instantly. As if it's a local response.
Could be something on my Pc?? It's an Ubuntu machine.

Spoiler: Pings via Wifi

# Router at IOT
SSH@ICX6450#ping 10.1.5.251
Sending 1, 16-byte ICMP Echo to 10.1.5.251, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 10.1.5.251      : bytes=16 time=1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.

# Router at CCTV
SSH@ICX6450#ping 10.1.6.5
Sending 1, 16-byte ICMP Echo to 10.1.6.5, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 10.1.6.5        : bytes=16 time=1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.

# My PC
SSH@ICX6450#ping 10.1.2.100
Sending 1, 16-byte ICMP Echo to 10.1.2.100, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 10.1.2.100      : bytes=16 time=1ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.

# 6450 from Router
root@opnsense:~ # ping 10.1.3.3
PING 10.1.3.3 (10.1.3.3): 56 data bytes
64 bytes from 10.1.3.3: icmp_seq=0 ttl=64 time=0.698 ms
64 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=0.600 ms
64 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=0.634 ms
^C
--- 10.1.3.3 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.600/0.638/0.698/0.036 ms

# My PC
root@opnsense:~ # ping 10.1.2.100
PING 10.1.2.100 (10.1.2.100): 56 data bytes
64 bytes from 10.1.2.100: icmp_seq=0 ttl=64 time=1.159 ms
64 bytes from 10.1.2.100: icmp_seq=1 ttl=64 time=0.928 ms
64 bytes from 10.1.2.100: icmp_seq=2 ttl=64 time=1.097 ms
^C
--- 10.1.2.100 ping statistics ---
11 packets transmitted, 11 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.928/1.255/2.251/0.377 ms

# NVR from my PC
$ ping 10.1.6.30
PING 10.1.6.30 (10.1.6.30) 56(84) bytes of data.
64 bytes from 10.1.6.30: icmp_seq=1 ttl=63 time=1.75 ms
64 bytes from 10.1.6.30: icmp_seq=2 ttl=63 time=1.61 ms
64 bytes from 10.1.6.30: icmp_seq=3 ttl=63 time=3.43 ms
64 bytes from 10.1.6.30: icmp_seq=4 ttl=63 time=1.52 ms
64 bytes from 10.1.6.30: icmp_seq=5 ttl=63 time=1.51 ms

--- 10.1.6.30 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 1.506/1.961/3.433/0.741 ms

Spoiler: Pings via RJ45

# From my PC
# 6450 Management Ip
$ ping 10.1.3.3
ping: connect: La red es inaccesible  #--->> The network is unreachable

# Router Ip on same VLAN/Segment
$ ping 10.1.2.251
PING 10.1.2.251 (10.1.2.251) 56(84) bytes of data.
64 bytes from 10.1.2.251: icmp_seq=1 ttl=64 time=0.131 ms
64 bytes from 10.1.2.251: icmp_seq=2 ttl=64 time=0.102 ms
64 bytes from 10.1.2.251: icmp_seq=3 ttl=64 time=0.104 ms
64 bytes from 10.1.2.251: icmp_seq=4 ttl=64 time=0.104 ms
64 bytes from 10.1.2.251: icmp_seq=5 ttl=64 time=0.133 ms

--- 10.1.2.251 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4095ms
rtt min/avg/max/mdev = 0.102/0.114/0.133/0.014 ms

# Router Ip on OTHER VLAN/Segment
$ ping 10.1.3.251
ping: connect: La red es inaccesible

# TPLink
$ ping 10.1.2.5
PING 10.1.2.5 (10.1.2.5) 56(84) bytes of data.
64 bytes from 10.1.2.5: icmp_seq=1 ttl=64 time=3.62 ms
64 bytes from 10.1.2.5: icmp_seq=2 ttl=64 time=1.26 ms
64 bytes from 10.1.2.5: icmp_seq=3 ttl=64 time=1.36 ms
64 bytes from 10.1.2.5: icmp_seq=4 ttl=64 time=1.30 ms
64 bytes from 10.1.2.5: icmp_seq=5 ttl=64 time=1.25 ms

--- 10.1.2.5 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 1.254/1.757/3.615/0.929 ms

# Wifi AP
$ ping 10.1.3.10
ping: connect: La red es inaccesible

 

[kapone]

If only I could see a basic configuration for a L3 switch + pure L2 switch downstream

I can help. That’s how my most recent configuration is, with the Mellanox SX6036 doing L3 routing, with a 6450 in essentially L2 mode with one minor thing. The management IP for it…sound familiar?

Not at home right now, will post later.

 

[SeRiusMe]

with the Mellanox SX6036 doing L3 routing

Wow! All those QSFP...

I've been changing the 6450 configuration. With the serial cable I managed to get the management interface arranged as I wanted.

Spoiler: ICX6450 sh run

SSH@ICX6450#sh run
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
  module 1 icx6450-24p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
  no legacy-inline-power
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 name LAN by port
tagged ethe 1/1/1 ethe 1/1/4 ethe 1/2/1 to 1/2/4
untagged ethe 1/1/3 ethe 1/1/24
!
vlan 3 name SYS by port
tagged ethe 1/1/1 ethe 1/1/4 ethe 1/2/1 to 1/2/4
untagged ethe 1/1/2
!
vlan 4 name WRX by port
tagged ethe 1/1/1 ethe 1/1/4 ethe 1/2/1 to 1/2/4
!
vlan 5 name IOT by port
tagged ethe 1/1/1 ethe 1/1/4 ethe 1/2/1 to 1/2/4
!
vlan 6 name CTV by port
tagged ethe 1/1/1 ethe 1/1/4 ethe 1/2/1 to 1/2/4
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname ICX6450
ip dhcp-client disable
ip dns server-address 10.1.3.251
ip route 0.0.0.0/0 10.1.3.251
!
no telnet server
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
disable serve
server 10.1.3.251
!
!
!
!
!
interface management 1
ip address 10.1.3.3 255.255.255.0
!
interface ethernet 1/1/1
port-name UPLINK
no spanning-tree
!
interface ethernet 1/1/2
port-name MGMT Jumper
no spanning-tree
!
interface ethernet 1/1/3
port-name PC Redcomet
no spanning-tree
!
interface ethernet 1/1/4
port-name AP Front
dual-mode  3
no spanning-tree
inline power
!
interface ethernet 1/1/24
port-name PR Laser
no spanning-tree
!
!
!
!
!
!
!
!
!
end

But I found something interesting in my PC.


It doesn't let me input the gateway for the ethernet interface.
And there's a suspicious virtual eth interface that I don't know from where it comes. Ans I can't bring it up.

Spoiler: PC Routes

# When connected via Wifi, the gateway is informed.
$ route -n
Tabla de rutas IP del núcleo
Destino         Pasarela        Genmask         Indic Métric Ref    Uso Interfaz
0.0.0.0         10.1.2.251      0.0.0.0         UG    600    0        0 wlp111s0
10.1.2.0        0.0.0.0         255.255.255.0   U     600    0        0 wlp111s0

# When connected via ethernet, it doens't have default route
$ route -n
Tabla de rutas IP del núcleo
Destino         Pasarela        Genmask         Indic Métric Ref    Uso Interfaz
10.1.2.0        0.0.0.0         255.255.255.0   U     100    0        0 enp0s31f6

EDIT: As soon as I add the default route, networking starts to work as expected.

$ sudo route add default gw 10.1.2.251 enp0s31f6

$ ping 10.1.3.12
PING 10.1.3.12 (10.1.3.12) 56(84) bytes of data.
64 bytes from 10.1.3.12: icmp_seq=1 ttl=63 time=0.706 ms
64 bytes from 10.1.3.12: icmp_seq=2 ttl=63 time=0.634 ms
64 bytes from 10.1.3.12: icmp_seq=3 ttl=63 time=0.638 ms
^C
--- 10.1.3.12 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2078ms
rtt min/avg/max/mdev = 0.634/0.659/0.706/0.033 ms

It would seem like the problem is my computer. Damn Kubuntu, defeating me again.
Removing the adapter and re-adding it solved the issue.

 

[kapone]

Wow! All those QSFP...

umm...yeah...lots of 40g ports...but...40g DAC cables are thick...and heavy...and quite inflexible...makes for a very untidy rack, and I don't like it.

This is not even fully populated yet, interim pic...had 8 more to go at that point.

 

[SeRiusMe]

I've started configuring the 6610 for Inter-vlan routing. At the moment it doesn't have anything connected to it, and initially it had all my current VLAN setup. Was connected to the old switch at port 1/1/2, all old vlans tagged.

Now I have configured a new set of VLANS and progressively removing the old. I added port 1/1/1 untagged to Transit VLAN and linked it to an empty interface of the opnsense. I followed Kapone's instructions for an inter-vlan setup and created the interface in opnsense, the gateway and added routes for all other new Vlans.
What is not clear to me is if I had to inform the gateway in the Transit interface itself or just create another route for it:

Problem 1 is that I don't have ping between router and switch in any direction. I've tried with or without the above setting.

Problem number two 2 is that I can't get rid of the last old Vlan in the 6610. Even when I've configured the management port with a valid Ip at the old management vlan and linked to an untagged port of the old switch, and that I'm connected via SSH to the switch at that mngt ip, as soon as I remove the ve interface on vlan 2, the switch loses ssh connection. Even vlan 2 is not on the same segment, and I've created an explicit route for it.

Spoiler: 6610 sh run

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
no legacy-inline-power
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 name LAN by port <<=== OLD VLAN
tagged ethe 1/1/2
router-interface ve 2
!
vlan 10 name Trusted by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 10
!
vlan 20 name Work by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 20
!
vlan 40 name CCTV by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 40
!
vlan 50 name IoT by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 50
!
vlan 172 name Management by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 172
!
vlan 192 name TRANSIT by port
untagged ethe 1/1/1 ethe 1/3/1
router-interface ve 192
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname ICX6610
ip dhcp-client disable
ip dns server-address 192.168.0.81
ip route 0.0.0.0/0 192.168.0.81
ip route 10.1.3.0/24 10.1.3.251 <<== EXPLICIT ROUTE
!
no telnet server
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
disable serve
server 192.168.0.81
!
!
interface management 1
ip address 10.1.3.1 255.255.255.0
!
interface ve 2 <<== If removed, comms stop
ip address 10.1.2.1 255.255.255.0
!
interface ve 10
ip address 10.10.10.1 255.255.255.0
!
interface ve 20
ip address 10.20.20.1 255.255.255.224
!
interface ve 40
ip address 10.40.40.1 255.255.254.0
!
interface ve 50
ip address 10.50.50.1 255.255.254.0
!
interface ve 172
ip address 172.22.0.1 255.255.255.0
!
interface ve 192
ip address 192.168.0.82 255.255.255.248

Shouldn't that config allow managing the switch only with the management Ip and be independent of that Vlan 2?

 

[fohdeesha]

I've started configuring the 6610 for Inter-valn routing. At the moment it doesn't have anything connected to it, and initially it had all my current VLAN setup. Was connected to the old switch at port 1/1/2, all old vlans tagged.

Now I have configured a new set of VLANS and progressively removing the old. I added port 1/1/1 untagged to Transit VLAN and linked it to an empty interface of the opnsense. I followed Kapone's instructions for an inter-vlan setup and created the interface in opnsense, the gateway and added routes for all other new Vlans.
What is not clear to me is if I had to inform the gateway in the Transit interface itself or just create another route for it:
View attachment 42742
Problem 1 is that I don't have ping between router and switch in any direction. I've tried with or without the above setting.

Problem number two 2 is that I can't get rid of the last old Vlan in the 6610. Even when I've configured the management port with a valid Ip at the old management vlan and linked to an untagged port of the old switch, and that I'm connected via SSH to the switch at that mngt ip, as soon as I remove the ve interface on vlan 2, the switch loses ssh connection. Even vlan 2 is not on the same segment, and I've created an explicit route for it.

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
no legacy-inline-power
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 name LAN by port <<=== OLD VLAN
tagged ethe 1/1/2
router-interface ve 2
!
vlan 10 name Trusted by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 10
!
vlan 20 name Work by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 20
!
vlan 40 name CCTV by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 40
!
vlan 50 name IoT by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 50
!
vlan 172 name Management by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 172
!
vlan 192 name TRANSIT by port
untagged ethe 1/1/1 ethe 1/3/1
router-interface ve 192
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname ICX6610
ip dhcp-client disable
ip dns server-address 192.168.0.81
ip route 0.0.0.0/0 192.168.0.81
ip route 10.1.3.0/24 10.1.3.251 <<== EXPLICIT ROUTE
!
no telnet server
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
disable serve
server 192.168.0.81
!
!
interface management 1
ip address 10.1.3.1 255.255.255.0
!
interface ve 2 <<== If removed, comms stop
ip address 10.1.2.1 255.255.255.0
!
interface ve 10
ip address 10.10.10.1 255.255.255.0
!
interface ve 20
ip address 10.20.20.1 255.255.255.224
!
interface ve 40
ip address 10.40.40.1 255.255.254.0
!
interface ve 50
ip address 10.50.50.1 255.255.254.0
!
interface ve 172
ip address 172.22.0.1 255.255.255.0
!
interface ve 192
ip address 192.168.0.82 255.255.255.248

Shouldn't that config allow managing the switch only with the management Ip and be independent of that Vlan 2?

dont specify a gateway in the actual interface page for that transit interface

also pings and shit aren't working probably because you haven't added any firewall rules for that transit interface, by default there won't be any so everything is denied. you'll need allow rules for the transit subnet itself and then any vlan subnets the switch will be routing, at the very least

 

[SeRiusMe]

dont specify a gateway in the actual interface page for that transit interface

also pings and shit aren't working probably because you haven't added any firewall rules for that transit interface, by default there won't be any so everything is denied. you'll need allow rules for the transit subnet itself and then any vlan subnets the switch will be routing, at the very least

Thank you fohdeesha for your help. I also take this opportunity to sincerely thank you for the work you've done on this thread, both initially and in keeping you active on it.

You where right, there wasn't any rule on the transit interface. I added the following:

Should then the router at least be able to ping the switch. But It doesn't.

This is the rest of my configuration:

Interface. Routes are also visible. 192.168.0.80/29 is created automatically:


Gateway:


What drives me crazy is not being able to directly ping or ssh to the management Ip 10.1.3.1 that I've rerouted through the old switch. It should travel independently from the new configuration.

 

[kapone]

I've started configuring the 6610 for Inter-vlan routing. At the moment it doesn't have anything connected to it, and initially it had all my current VLAN setup. Was connected to the old switch at port 1/1/2, all old vlans tagged.

Now I have configured a new set of VLANS and progressively removing the old. I added port 1/1/1 untagged to Transit VLAN and linked it to an empty interface of the opnsense. I followed Kapone's instructions for an inter-vlan setup and created the interface in opnsense, the gateway and added routes for all other new Vlans.
What is not clear to me is if I had to inform the gateway in the Transit interface itself or just create another route for it:
View attachment 42742
Problem 1 is that I don't have ping between router and switch in any direction. I've tried with or without the above setting.

Problem number two 2 is that I can't get rid of the last old Vlan in the 6610. Even when I've configured the management port with a valid Ip at the old management vlan and linked to an untagged port of the old switch, and that I'm connected via SSH to the switch at that mngt ip, as soon as I remove the ve interface on vlan 2, the switch loses ssh connection. Even vlan 2 is not on the same segment, and I've created an explicit route for it.

Spoiler: 6610 sh run

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
no legacy-inline-power
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 2 name LAN by port <<=== OLD VLAN
tagged ethe 1/1/2
router-interface ve 2
!
vlan 10 name Trusted by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 10
!
vlan 20 name Work by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 20
!
vlan 40 name CCTV by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 40
!
vlan 50 name IoT by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 50
!
vlan 172 name Management by port
tagged ethe 1/1/3 ethe 1/3/3
router-interface ve 172
!
vlan 192 name TRANSIT by port
untagged ethe 1/1/1 ethe 1/3/1
router-interface ve 192
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname ICX6610
ip dhcp-client disable
ip dns server-address 192.168.0.81
ip route 0.0.0.0/0 192.168.0.81
ip route 10.1.3.0/24 10.1.3.251 <<== EXPLICIT ROUTE
!
no telnet server
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
disable serve
server 192.168.0.81
!
!
interface management 1
ip address 10.1.3.1 255.255.255.0
!
interface ve 2 <<== If removed, comms stop
ip address 10.1.2.1 255.255.255.0
!
interface ve 10
ip address 10.10.10.1 255.255.255.0
!
interface ve 20
ip address 10.20.20.1 255.255.255.224
!
interface ve 40
ip address 10.40.40.1 255.255.254.0
!
interface ve 50
ip address 10.50.50.1 255.255.254.0
!
interface ve 172
ip address 172.22.0.1 255.255.255.0
!
interface ve 192
ip address 192.168.0.82 255.255.255.248

Shouldn't that config allow managing the switch only with the management Ip and be independent of that Vlan 2?

You're making this harder than it needs to be. Forget about the management IP etc for now. Connect the console cable and a serial session, and first get the basic things working.

1. Why do you have TWO ports on the 6610 for your transit network? You only need one, unless you're doing a LAG, which is a whole another issue/configuration.
2. Your IP address for the TRANSIT interface on the 6610 is 192.168.0.82 and on the Opnsense end is 192.168.0.81, correct? (Weird choice of IPs, but ok...)
3. @fohdeesha is right, don't specify a gateway for that interface, you define the Gateway separately and choose the interface while creating the gateway.
4. (My apologies, I didn't mention the firewall rules in my post on that thread).
5. Your firewall rule is incorrect. Your protocol needs to be IPv4 ICMP, not IPv4. That's why ping is not working. Once you get this right, ping will still work only between the switch and firewall.
6. Don't forget to add static routes in Opnsense for all CIDR ranges that you expect to flow between the switch and firewall. The gateway for all these routes should be the one you defined above.
7. Once you start creating firewall rules for that traffic, make sure you use "Network/Host" as the source, not an interface.

 

[SeRiusMe]

Forget about the management IP etc for now.

Fixed. It was a bad cable/patch panel port. Now I've been able to remove that remaining ve 2. If I was been able to get to the switch via the old Vlan 2, there should had been some kind of network loop. I have STP disabled, so who knows what was happening.

Spoiler: Current cleaned config

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
stack disable
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 10 name Trusted by port
 tagged ethe 1/1/3 ethe 1/3/3
 router-interface ve 10
!
vlan 20 name Work by port
 tagged ethe 1/1/3 ethe 1/3/3
 router-interface ve 20
!
vlan 40 name CCTV by port
 tagged ethe 1/1/3 ethe 1/3/3
 router-interface ve 40
!
vlan 50 name IoT by port
 tagged ethe 1/1/3 ethe 1/3/3
 router-interface ve 50
!
vlan 172 name Management by port
 tagged ethe 1/1/3 ethe 1/3/3
 router-interface ve 172
!
vlan 192 name TRANSIT by port
 untagged ethe 1/1/1 ethe 1/3/1
 router-interface ve 192
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname ICX6610
ip dhcp-client disable
ip dns server-address 192.168.0.81
ip route 0.0.0.0/0 192.168.0.81
!
no telnet server
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
 disable serve
 server 192.168.0.81
!
!
interface management 1
 ip address 10.1.3.1 255.255.255.0
!
interface ve 10
 ip address 10.10.10.1 255.255.255.0
!
interface ve 20
 ip address 10.20.20.1 255.255.255.224
!
interface ve 40
 ip address 10.40.40.1 255.255.254.0
!
interface ve 50
 ip address 10.50.50.1 255.255.254.0
!
interface ve 172
 ip address 172.22.0.1 255.255.255.0
!
interface ve 192
 ip address 192.168.0.82 255.255.255.248
!
end

1. Why do you have TWO ports on the 6610 for your transit network?

I also configured a SFP+ link, in case there wasn't any RJ45 port free on the firewall. It's not connected ATM and I don't think I would connect it. The SFP+ card on the fw may come to my station.

2. Your IP address for the TRANSIT interface on the 6610 is 192.168.0.82 and on the Opnsense end is 192.168.0.81, correct?

Yep! It's weird. I thought the segment was too obvious for that interface. Dunno if it may make a difference in case of an intrusion. Just thought, why not?

3. @fohdeesha is right

As always
So you think the current firewall configuration is right?

4. (My apologies

C'mon, it's my fault. I remember you mentioning it on another thread. But I'm still not sure how these rules should be made.

5. Your firewall rule is incorrect.

I think you'r wrong there. Look at the asterisk near the IPv4. Says all protocols at IPv4.

The pings between the FW and the switch are working now in both directions. It was that Management cable and VE 2 doing havoc.

6. Don't forget to add static routes in Opnsense for all CIDR ranges that you expect to flow between the switch and firewall. The gateway for all these routes should be the one you defined above.

Like this? Missing anything?

Note there are two networks in /23 subnet. My intention is putting all the devices in, for example 10.50.51.X and deny internet to all the segment.
Then the devices in 10.50.50.X should access internet and all of them should talk together. I guess I want to simplify the future ACLs.

7. Once you start creating firewall rules for that traffic, make sure you use "Network/Host" as the source, not an interface.

Could you please give me some examples of those rules? Aside from the rule I posted before, Do I need something more?
I'm used to the normal behavior of only one allow all per interface simply does work.
Do I need a 10.10.10.0/24 Allow.... All? rule in the Transit interface? For Inet? That seems to me that would allow routing to other vlans coming from the FW.

In the end, It should only remain the Transit interface and the Wan interface, isn't it?