Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

rootpeer

Member
Oct 19, 2019
73
13
8
I have a weird issue and I need some help.

I have two ICX6450 switches connected to each other.
SW1 has VLAN1 on 10.1.0.0/24 and SW2 has VLAN1 on 10.2.0.0/24.
Both switches are connected via VLAN1012 on 10.1.2.0/24.
Both switches have ve's on VLAN1012 and VLAN1.
Both switches have static routes configured for each other's VLAN1 subnet via their "partner's" VLAN1012 ve. I will post the config below.

I am trying to route from SW1 VLAN1 to SW2 VLAN2. Using a host on SW1, I can ping some hosts on SW2 but not others. Traceroute does not help at all. The host is configured with a static IP and the SW1 VLAN1 ve IP as its gateway for this troubleshooting session.

SW1:
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A 609c.9f79.4b20                 
ve1012     Up      N/A     N/A  N/A   None  N/A N/A  N/A 609c.9f79.4b20   

###################################################

interface ve 1                                                   
 ip address 10.1.0.2 255.255.255.0
!
interface ve 1012
 ip address 10.1.2.240 255.255.255.0

################################################

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          10.1.0.1        ve 1          1/1           S    6d11h
2       10.1.0.0/24        DIRECT          ve 1          0/0           D    6d11h
3       10.1.2.0/24        DIRECT          ve 1012       0/0           D    5d21h
4       10.2.0.0/24        10.1.2.241      ve 1012       1/1           S    5d21h

SW2:
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2465.ea40                 
ve1012     Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2465.ea40

#################################################

interface ve 1
 ip address 10.2.0.2 255.255.255.0
!
interface ve 1012
 ip address 10.1.2.241 255.255.255.0

#################################################

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          10.2.0.1        ve 1          1/1           S    27m30s
2       10.1.0.0/24        10.1.2.240      ve 1012       1/1           S    5d21h
3       10.1.2.0/24        DIRECT          ve 1012       0/0           D    7d0h 
4       10.2.0.0/24        DIRECT          ve 1          0/0           D    25d4h


traceroute to reachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.20
traceroute to 10.2.0.20 (10.2.0.20), 30 hops max, 60 byte packets
 1  _gateway (10.1.0.2)  1.420 ms  2.451 ms  1.737 ms
 2  10.1.2.241 (10.1.2.241)  219.946 ms  219.910 ms  219.886 ms
 3  10.2.0.20 (10.2.0.20)  219.929 ms  219.911 ms  219.877 ms
traceroute to an unreachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.38
traceroute to 10.2.0.38 (10.2.0.38), 30 hops max, 60 byte packets
 1  _gateway (10.1.0.2)  1.229 ms  2.380 ms  1.677 ms
 2  10.1.2.241 (10.1.2.241)  2.658 ms  8.522 ms  8.492 ms
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
traceroute to a second unreachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.4
traceroute to 10.2.0.4 (10.2.0.4), 30 hops max, 60 byte packets
 1  _gateway (10.1.0.2)  7.037 ms  7.712 ms  8.369 ms
 2  10.1.2.241 (10.1.2.241)  5.443 ms  5.417 ms  5.392 ms
 3  10.1.2.116 (10.1.2.116)  6.600 ms  6.576 ms  6.550 ms
This last one is particularly confusing to me because 10.1.2.116 is a pfSense interface on VLAN1012 at SW2 location. Why is the SW2 trying to forward the packet to the pfSense interface instead of the actual host at 10.2.0.4?

Please help!
 

Craig Curtin

Member
Jun 18, 2017
101
20
18
59
I have a weird issue and I need some help.

I have two ICX6450 switches connected to each other.
SW1 has VLAN1 on 10.1.0.0/24 and SW2 has VLAN1 on 10.2.0.0/24.
Both switches are connected via VLAN1012 on 10.1.2.0/24.
Both switches have ve's on VLAN1012 and VLAN1.
Both switches have static routes configured for each other's VLAN1 subnet via their "partner's" VLAN1012 ve. I will post the config below.

I am trying to route from SW1 VLAN1 to SW2 VLAN2. Using a host on SW1, I can ping some hosts on SW2 but not others. Traceroute does not help at all. The host is configured with a static IP and the SW1 VLAN1 ve IP as its gateway for this troubleshooting session.

SW1:
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A 609c.9f79.4b20                
ve1012     Up      N/A     N/A  N/A   None  N/A N/A  N/A 609c.9f79.4b20  

###################################################

interface ve 1                                                  
ip address 10.1.0.2 255.255.255.0
!
interface ve 1012
ip address 10.1.2.240 255.255.255.0

################################################

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          10.1.0.1        ve 1          1/1           S    6d11h
2       10.1.0.0/24        DIRECT          ve 1          0/0           D    6d11h
3       10.1.2.0/24        DIRECT          ve 1012       0/0           D    5d21h
4       10.2.0.0/24        10.1.2.241      ve 1012       1/1           S    5d21h

SW2:
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2465.ea40                
ve1012     Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2465.ea40

#################################################

interface ve 1
ip address 10.2.0.2 255.255.255.0
!
interface ve 1012
ip address 10.1.2.241 255.255.255.0

#################################################

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          10.2.0.1        ve 1          1/1           S    27m30s
2       10.1.0.0/24        10.1.2.240      ve 1012       1/1           S    5d21h
3       10.1.2.0/24        DIRECT          ve 1012       0/0           D    7d0h
4       10.2.0.0/24        DIRECT          ve 1          0/0           D    25d4h


traceroute to reachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.20
traceroute to 10.2.0.20 (10.2.0.20), 30 hops max, 60 byte packets
1  _gateway (10.1.0.2)  1.420 ms  2.451 ms  1.737 ms
2  10.1.2.241 (10.1.2.241)  219.946 ms  219.910 ms  219.886 ms
3  10.2.0.20 (10.2.0.20)  219.929 ms  219.911 ms  219.877 ms
traceroute to an unreachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.38
traceroute to 10.2.0.38 (10.2.0.38), 30 hops max, 60 byte packets
1  _gateway (10.1.0.2)  1.229 ms  2.380 ms  1.677 ms
2  10.1.2.241 (10.1.2.241)  2.658 ms  8.522 ms  8.492 ms
3  * * *
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
traceroute to a second unreachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.4
traceroute to 10.2.0.4 (10.2.0.4), 30 hops max, 60 byte packets
1  _gateway (10.1.0.2)  7.037 ms  7.712 ms  8.369 ms
2  10.1.2.241 (10.1.2.241)  5.443 ms  5.417 ms  5.392 ms
3  10.1.2.116 (10.1.2.116)  6.600 ms  6.576 ms  6.550 ms
This last one is particularly confusing to me because 10.1.2.116 is a pfSense interface on VLAN1012 at SW2 location. Why is the SW2 trying to forward the packet to the pfSense interface instead of the actual host at 10.2.0.4?

Please help!
I am no expert on the Brocade - but having different IP subnets on the same VLAN is just asking for trouble in other implementations - not sure why you would try and do that as even if it does work it is just asking for confusion

Personally i never use the VLAN 1 (default VLAN) for any traffic and change it to a different number as part of my switch setup

Craig
 

rootpeer

Member
Oct 19, 2019
73
13
8
I am no expert on the Brocade - but having different IP subnets on the same VLAN is just asking for trouble in other implementations - not sure why you would try and do that as even if it does work it is just asking for confusion

Personally i never use the VLAN 1 (default VLAN) for any traffic and change it to a different number as part of my switch setup

Craig
The switches are at separate locations connected via a pair of Ubiquiti antennas. The two VLAN1s are not connected at L2, thus the need to route between them through VLAN1012 at L3. So, even if they both have the same VLAN tag, they are two separate networks. I don't see how them having the same tag is relevant here, they just need to forward the packets to the next hop that is statically assigned.
 

vangoose

Active Member
May 21, 2019
326
104
43
Canada
The switches are at separate locations connected via a pair of Ubiquiti antennas. The two VLAN1s are not connected at L2, thus the need to route between them through VLAN1012 at L3. So, even if they both have the same VLAN tag, they are two separate networks. I don't see how them having the same tag is relevant here, they just need to forward the packets to the next hop that is statically assigned.
You just need to configure route on each switch to so it knows where to forward traffics to. Your client only needs 1 default gateway, no need to insert custom routes.
 

ramicio

Member
Nov 30, 2022
69
14
8
Hello. I recently got an ICX-6610-24p switch, and I am having trouble with the 40g ports. I bought an Intel XL710-QDA1 NIC, and my QSFP+ DAC cable is one from FS, just "generic." I am using Ubuntu Server 20.04. I did the tutorial for licensing and whatnot and everything seems to work. 10g front ports work. On the back I have the QSFP+ cable plugged into the top left of the 4 ports. I cannot get an IP address via DHCP. It, however, does show up as a client on my router (where I have static DHCP stuff set). Pings to and fro just fine if I use an internal IP address. Go to ping a site like google, and it can't even get any DNS record for that. I updated the firmware of the NIC. No change. I can set it to a static IP address and no change. ifconfig shows almost as many RX errors as there are RX packets. None on the TX side of things. If I plug it into the breakout ports, nothing (even though I followed the tutorial and that's not in the config anymore). If I try the other proper 40g port (bottom left), I get no link, whatsoever. Should I be barking up the tree to the switch seller (fried ports), or is there more I should be trying?

Thank you.
 

dtremit

New Member
Aug 20, 2018
6
2
3
Quick question for those more familiar with these switches than I — if I need to change from the switch image to the router image, can I safely do so from the normal config prompt copy command, and then reboot (rather than going through the boot prompt)?

I configured my ICX6450 ages ago following @fohdeesha 's excellent guide, and I thought I had used the router version of the firmware — but it looks like at least my secondary image is the switch version:

Code:
SSH@icx6450#show flash
Stack unit 1:
  Compressed Pri Code size = 9871112, Version:08.0.30uT313 (primary)
  Compressed Sec Code size = 8526668, Version:08.0.30kT311 (ICX64S08030k.bin)
  Compressed Boot-Monitor Image size = 786944, Version:10.1.05T310
  Code Flash Free Space = 32514048
And indeed, I don't seem to be able to get some of the L3 features to work...
 

rootpeer

Member
Oct 19, 2019
73
13
8
Hello. I recently got an ICX-6610-24p switch, and I am having trouble with the 40g ports. I bought an Intel XL710-QDA1 NIC, and my QSFP+ DAC cable is one from FS, just "generic." I am using Ubuntu Server 20.04. I did the tutorial for licensing and whatnot and everything seems to work. 10g front ports work. On the back I have the QSFP+ cable plugged into the top left of the 4 ports. I cannot get an IP address via DHCP. It, however, does show up as a client on my router (where I have static DHCP stuff set). Pings to and fro just fine if I use an internal IP address. Go to ping a site like google, and it can't even get any DNS record for that. I updated the firmware of the NIC. No change. I can set it to a static IP address and no change. ifconfig shows almost as many RX errors as there are RX packets. None on the TX side of things. If I plug it into the breakout ports, nothing (even though I followed the tutorial and that's not in the config anymore). If I try the other proper 40g port (bottom left), I get no link, whatsoever. Should I be barking up the tree to the switch seller (fried ports), or is there more I should be trying?

Thank you.
It doesn't sound like this is your problem but might be worth investigating. The Intel NICs don't like modules that are not Intel branded. There is a kernel module option to disable this with SFP+ cards but not with the X710. Have a look at this before you start troubleshooting the switch: https://forum.vyos.io/t/unsupported-sfp-transceivers-on-intel-nic/6923/3
 

ramicio

Member
Nov 30, 2022
69
14
8
It doesn't sound like this is your problem but might be worth investigating. The Intel NICs don't like modules that are not Intel branded. There is a kernel module option to disable this with SFP+ cards but not with the X710. Have a look at this before you start troubleshooting the switch: Unsupported SFP+ transceivers on Intel NIC
The thing is there is packet flow when plugged into the "top left" port (on the back of the Brocade). The packet flow is nothing but errors. But if I plug the DAC into the port below (bottom left) it, no link is being established, whatsoever. No lights on the NIC, nothing. Also, removed the whole "breakout" thing from the config for the other 2 ports, and they act the same as the "bottom left" port. Nothing, whatsoever from them. It's a cable, not an optical module. Ethtool shows it as "40000baseCR4/Full." So, the top-left port is the only one that has signs of life, and it's just nothing but errors. No other ports show any signs of life.

Tried the link. Did make. No luck. Just spits out "ioctl: Invalid argument"
 
  • Like
Reactions: rootpeer

ramicio

Member
Nov 30, 2022
69
14
8
Had to go into the c file and change 1572 to 1584. No idea about any of the offsets. I see stuff about that all over, and I have no idea where they're getting the numbers, and I'm not a programmer, whatsoever, so I have no idea what any of that means. All I know is if I run that tool again it says it's unlocked. No change in results.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Quick question for those more familiar with these switches than I — if I need to change from the switch image to the router image, can I safely do so from the normal config prompt copy command, and then reboot (rather than going through the boot prompt)?

I configured my ICX6450 ages ago following @fohdeesha 's excellent guide, and I thought I had used the router version of the firmware — but it looks like at least my secondary image is the switch version:

Code:
SSH@icx6450#show flash
Stack unit 1:
  Compressed Pri Code size = 9871112, Version:08.0.30uT313 (primary)
  Compressed Sec Code size = 8526668, Version:08.0.30kT311 (ICX64S08030k.bin)
  Compressed Boot-Monitor Image size = 786944, Version:10.1.05T310
  Code Flash Free Space = 32514048
And indeed, I don't seem to be able to get some of the L3 features to work...
it doesn't matter what's in the secondary slot because by default it boots from the primary, it will only boot from the secondary if the primary is corrupted or you explicitly tell it to. What's your output of "show version" and what l3 features are you having issues with?
 

Mailkit

New Member
Nov 6, 2017
20
5
3
78
Hello,

Is it possible to change the welcome screen to my liking and remove brocade name?

Thanking you
 

dtremit

New Member
Aug 20, 2018
6
2
3
it doesn't matter what's in the secondary slot because by default it boots from the primary, it will only boot from the secondary if the primary is corrupted or you explicitly tell it to. What's your output of "show version" and what l3 features are you having issues with?
Thanks for the quick reply — and for your guide, which was really invaluable! (Very long-time lurker here, I'm afraid.)

EDIT — realized after I posted this that I did in fact miss something in my config — I set up the router-interface for all of my new VLANs, but not the existing one they'll use to get outbound. D'oh.
 
Last edited:
  • Like
Reactions: fohdeesha

Cobra0101

New Member
Nov 22, 2022
10
0
1
what wrong with my running config, please? Cant connect to my web management, or if connect to my switch my router management but can ping to external site from the router now. Running in router mode



!
aaa authentication snmp-server default local
aaa authentication web-server default local
aaa authentication login default local
ip default-network 10.0.0.0/23
ip dns server-address 10.0.0.1 10.0.0.5
ip route 0.0.0.0/0 10.0.0.1
!
username *** password .....
!
!
clock timezone us Alaska
!
!
ntp
server 134.0.16.1 minpoll 5
server 162.159.200.1
server 217.114.59.66
server 185.83.169.27
!
!
web-management enable ethe 1/1/1
!
interface ethernet 1/1/1
ip address 10.0.0.4 255.255.255.0
!
!
end
 
Last edited:

Craig Curtin

Member
Jun 18, 2017
101
20
18
59
Hello. I recently got an ICX-6610-24p switch, and I am having trouble with the 40g ports. I bought an Intel XL710-QDA1 NIC, and my QSFP+ DAC cable is one from FS, just "generic." I am using Ubuntu Server 20.04. I did the tutorial for licensing and whatnot and everything seems to work. 10g front ports work. On the back I have the QSFP+ cable plugged into the top left of the 4 ports. I cannot get an IP address via DHCP. It, however, does show up as a client on my router (where I have static DHCP stuff set). Pings to and fro just fine if I use an internal IP address. Go to ping a site like google, and it can't even get any DNS record for that. I updated the firmware of the NIC. No change. I can set it to a static IP address and no change. ifconfig shows almost as many RX errors as there are RX packets. None on the TX side of things. If I plug it into the breakout ports, nothing (even though I followed the tutorial and that's not in the config anymore). If I try the other proper 40g port (bottom left), I get no link, whatsoever. Should I be barking up the tree to the switch seller (fried ports), or is there more I should be trying?

Thank you.
You need to post up your config and also provide a little more detail - are you saying you are connecting the Intel card to the breakout cable - what is then mean to be providing DHCP to the Intel card and which switch port is that connected to ?

Craig
 

Craig Curtin

Member
Jun 18, 2017
101
20
18
59
what wrong with my running config, please? Cant connect to my web management, or if connect to my switch my router management but can ping to external site from the router now. Running in router mode



!
aaa authentication snmp-server default local
aaa authentication web-server default local
aaa authentication login default local
ip default-network 10.0.0.0/23
ip dns server-address 10.0.0.1 10.0.0.5
ip route 0.0.0.0/0 10.0.0.1
!
username *** password .....
!
!
clock timezone us Alaska
!
!
ntp
server 134.0.16.1 minpoll 5
server 162.159.200.1
server 217.114.59.66
server 185.83.169.27
!
!
web-management enable ethe 1/1/1
!
interface ethernet 1/1/1
ip address 10.0.0.4 255.255.255.0
!
!
end
What code are you running on the switch ?

What VLANs have you defined ? - none in the config you have posted.

What is this switch ?

Craig
 

ramicio

Member
Nov 30, 2022
69
14
8
You need to post up your config and also provide a little more detail - are you saying you are connecting the Intel card to the breakout cable - what is then mean to be providing DHCP to the Intel card and which switch port is that connected to ?

Craig
Hello. I don't have very much understanding of this stuff, so if you would be able to tell me exactly what I need to do to be able to give you this information, that would be appreciated.

No breakout cable. A QSFP+ DAC. Only 1 port on the switch shows any sign of electrical life.

Means of providing DHCP? Probably my router. Connected to one of the regular RJ45 ports. Doesn't matter. If I use a static IP it's nothing but trouble.
 

gregsachs

Active Member
Aug 14, 2018
559
192
43
Interpreting TDR results?
(Disclaimer: I"m just missing somethin stupid)
Ok, I have a cable running to a closet, USW-Flex in there to forward PoE to an AP and such.
One cable run downstream of the Flex will only link at 100M, not 1GB. If I jumper around the USW and have the 6450 patched to that run, a TDR test tells me to look at local pair B.
I've looked here:
and here:
and for the life of me I can't decide if that means orange or green, given all my keyholes are wired 568B.
Just trying to figure out where to focus.