Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
@sash you say you're not using VEs on the brocade side, and that you assigned an IP directly to port 1/3/1 (which I wouldn't recommend, especially in an l3 config, but it's up to you), but then post a big list of VEs ?

I would figure that out, and then try the ping command specifying a source interface to use, so you at least know what IP the ICX is using to try and ping with:

ping 8.8.8.8 source 192.168.1.1
 

cserve

New Member
Feb 17, 2019
2
0
1
So what I am going to do with mine is stack across the breakout ports, and use the 4 total ports across my 2x 6610's for 40g.
Hello all , I'm trying to do the same without success, using just one breakout port (an entire 4x 10G connect via QSFP DAC) for stacking across a pair of 6610 and leave the all 40G ports to connect to my servers as well as the other breakout port free.

I have tried to remove all stack-trunks but I cannot select 1/2/2 as a stack port. It errors out saying only 1/2/1 and 1/2/6 can be use as stacking ports which are the exact 40G port I was trying to leave free.

The stack works with the stack-trunk 1/2/1 to 1/2/2, stack-port 1/2/1 and only connects the breakout port 1/2/2 via QSFP DAC. However 1/2/1 is still tied up with the stack-trunk and cannot be used to connect to servers for the 40G connection.

Is this possible? I have searched the entire thread and did not see a configuration for this. I found an opposite configuration from fohdeesha where stacking is done on all the 40G ports and leave the breakout ports free.

Thanks
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Hello all , I'm trying to do the same without success, using just one breakout port (an entire 4x 10G connect via QSFP DAC) for stacking across a pair of 6610 and leave the all 40G ports to connect to my servers as well as the other breakout port free.

I have tried to remove all stack-trunks but I cannot select 1/2/2 as a stack port. It errors out saying only 1/2/1 and 1/2/6 can be use as stacking ports which are the exact 40G port I was trying to leave free.

The stack works with the stack-trunk 1/2/1 to 1/2/2, stack-port 1/2/1 and only connects the breakout port 1/2/2 via QSFP DAC. However 1/2/1 is still tied up with the stack-trunk and cannot be used to connect to servers for the 40G connection.

Is this possible? I have searched the entire thread and did not see a configuration for this. I found an opposite configuration where stacking is done on all the 40G ports and leave the breakout ports free.

I'm not sure if that's possible, but given that very specific error message, it looks like it's not. I'm running a bit of a mix of that and the 40g ports on my home stack, it's stacked via a trunk of 1 40g port and 1 breakout port, leaving me 2x 40gbE ports and 2x breakout ports

it's also worth noting stack traffic over the 4x 10gbE QSFP+ ports is hashed similar to standard LACP, eg by source IP / port. Meaning if you stack using only the breakout ports, you won't get single-stream speed above 10gbps between stuff connected to each stack member. so 40gbE host on switch 1 is not going to be able to do more than 10gbps for single stream stuff (eg something using only one source IP and one source port, like SMB shares etc (unless running SMB 3.0)) to a 40gbE device on switch 2, as it will get hashed to just 1 of the 10gbps stacking channels
 
Last edited:

sash

Member
Nov 22, 2019
44
8
8
@sash you say you're not using VEs on the brocade side, and that you assigned an IP directly to port 1/3/1 (which I wouldn't recommend, especially in an l3 config, but it's up to you), but then post a big list of VEs ?

I would figure that out, and then try the ping command specifying a source interface to use, so you at least know what IP the ICX is using to try and ping with:

ping 8.8.8.8 source 192.168.1.1
I tried to mimic the old Cisco switch config where I used a no switchport command on the interface to turn it to L3 interface and assigned IP address to it directly. What is the downside of assigning IP directly to the interface and not to virtual interface?

I have tried pinging with source address included and all vlans that are configured with ospf routing can access the internet without issues.

Code:
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.0.11
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=15ms TTL=56
Success rate is 100 percent (1/1), round-trip min/avg/max=15/15/15 ms.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.3.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=14ms TTL=56
Success rate is 100 percent (1/1), round-trip min/avg/max=14/14/14 ms.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.11.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.12.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=16ms TTL=55
Success rate is 100 percent (1/1), round-trip min/avg/max=16/16/16 ms.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.35.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=14ms TTL=56
Success rate is 100 percent (1/1), round-trip min/avg/max=14/14/14 ms.
SSH@ICX6610-48P#ping 8.8.8.8 source 10.0.0.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.254.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
Here is ospf configuration on the Edge router:
Code:
 protocols {
     ospf {
         area 0.0.0.0 {
             area-type {
                 normal
             }
             network 172.16.0.0/16
             network 192.168.29.0/24
         }
         log-adjacency-changes {
         }
         parameters {
             abr-type cisco
             router-id 91.200.xxx.xxx
         }
     }
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
it will work (IP directly on switch port), but it's just much less flexible if you ever want to move anything around or create complex ACLs etc

judging from your results, it sounds like you haven't manually added static routes to the edge router for the non-ospf subnets on the ICX. that's why OSPF interfaces can get out to the internet, but the others can't. The ICX can get out to whatever, OSPF or not, as it has a default static route to the edgerouter (192.168.29.254), so the traffic is all being routed correctly from the ICX.

However when the edgerouter tries to send the ICMP response back to the ICX, like to address 192.168.11.1, it looks in its routing table to see what IP next-hop to send that packet to, and it has no entries, as they haven't been sent by the ICX via ospf, and you haven't entered one manually. If you have a default route on the edgerouter (like out to the internet), it will fall back to using this route as it couldn't find anything more specific, and the ICMP reply meant for the ICX will shoot back out to the internet. You can confirm this by running "show ip route 192.168.11.1" on the edgerouter, I bet it'll return your default internet gateway

The solution is to either have the ICX send the edgerouter the proper return routes via ospf by enabling ospf on these subnets as well, or entering static routes on the edgerouter for these non-ospf subnets. I'm not an ubiquiti user, but from a quick google, you would run the below to add the static routes for the non-ospf subnets

Code:
configure
set protocols static route 192.168.11.0/24 next-hop 192.168.29.1
set protocols static route 10.0.0.0/8 next-hop 192.168.29.1
etc
commit;save
 
Last edited:

sash

Member
Nov 22, 2019
44
8
8
(just fixed a couple typos on the edgerouter commands, if you copy/pasted them you might want to re-check)
My problem is different. I would like ICX itself to access the internet. Vlans 11 and 100 do not need to access internet by design. I'm routing everything through OSPF. I've added vlans to osfp routing and they work just fine. The routed interface on the ICX is also added to OSPF but it cannot access the internet. That is what I'm trying to figure out...

Code:
!
interface ethernet 1/3/1
 port-name Uplink to EdgeRoute6 on port eth5
 ip address 192.168.29.1 255.255.255.0
 ip ospf area 0
!
!
router ospf
 area 0
 neighbor 192.168.29.254
!
ip route 0.0.0.0/0 192.168.29.254
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
My problem is different. I would like ICX itself to access the internet. Vlans 11 and 100 do not need to access internet by design
that doesn't change the issue, the ICX's IP address in those vlans is still in those subnets, and the edgerouter has no idea how to get packets back to those subnets. If you want to just allow traffic back to the ICX IPs themselves but not the rest of the subnet, it would be

Code:
configure
set protocols static route 192.168.11.1/32 next-hop 192.168.29.1
set protocols static route 10.0.0.1/32 next-hop 192.168.29.1
etc
commit;save
if you mean to say you ran "ping 8.8.8.8 source 192.168.29.1" (using the l3 interface connected to the edgerouter as the source) and it failed, that's another story, as you have indeed enabled ospf on that interface, but that shouldn't even matter, as the edgerouter has an interface directly in this subnet (192.168.29.254) so it will have a route to this subnet already automatically (unless it's netmask is misconfigured or something). If this ping command is indeed failing, post the output of "show ip route 192.168.29.1" from the edgerouter. I doubt that ping command will fail though, if the edgerouter didn't have a route back to 192.168.29.1, none of the ospf enabled subnets would work (and they do)
 

sash

Member
Nov 22, 2019
44
8
8
that doesn't change the issue, the ICX's IP address in those vlans is still in those subnets, and the edgerouter has no idea how to get packets back to those subnets. If you want to just allow traffic back to the ICX IPs themselves but not the rest of the subnet, it would be

Code:
configure
set protocols static route 192.168.11.1/32 next-hop 192.168.29.1
set protocols static route 10.0.0.1/32 next-hop 192.168.29.1
etc
commit;save
if you mean to say you ran "ping 8.8.8.8 source 192.168.29.1" (using the l3 interface connected to the edgerouter as the source) and it failed, that's another story, as you have indeed enabled ospf on that interface, but that shouldn't even matter, as the edgerouter has an interface directly in this subnet (192.168.29.254) so it will have a route to this subnet already automatically (unless it's netmask is misconfigured or something). If this ping command is indeed failing, post the output of "show ip route 192.168.29.1" from the edgerouter. I doubt that ping command will fail though, if the edgerouter didn't have a route back to 192.168.29.1, none of the ospf enabled subnets would work (and they do)
I don't need internet for those two subnets - they are for storage connections inside my lan, thus I did not include them into OSPF routing. Router does not need to know about them at all.
The only problem is internet access for ICX itself. ICX can access the router on the directly connected interface. It can also access remote VPN subnets without issues. But it cannot access the internet.

Code:
SSH@ICX6610-48P#ping 8.8.8.8
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#ping 192.168.29.254 (EDGE ROUTER)
Sending 1, 16-byte ICMP Echo to 192.168.29.254, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.29.254  : bytes=16 time<1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.
SSH@ICX6610-48P#ping 192.168.7.1 (REMOVE SUBNET)
Sending 1, 16-byte ICMP Echo to 192.168.7.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.7.1     : bytes=16 time=14ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=14/14/14 ms.
SSH@ICX6610-48P#ping 192.168.23.1 (REMOTE SUBNET)
Sending 1, 16-byte ICMP Echo to 192.168.23.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.23.1    : bytes=16 time=3ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=3/3/3 ms.
 

CED6688

New Member
Dec 4, 2019
15
10
3
Looks like a NAT issue on the edgerouter. I seem to recall that the default edgerouter config only NATs the first LAN port.
 

dashpuppy

Member
Dec 16, 2018
48
21
8
Anyone know why the brocade switches (icx-6430-C12 ) would have issues displaying the web gui ? It's enable and have setup the user & aaa authentication login default local aaa authentication enable default local aaa authentication web default local commands but it only shows the port display. I can't get to any configuration menu's.

Bought a pair of them for Christmas, I have it up and running just having some little issues with vlans so i wanted to log into the GUI and see what i might be missing.

TIA !
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
I don't need internet for those two subnets - they are for storage connections inside my lan, thus I did not include them into OSPF routing. Router does not need to know about them at all.
The only problem is internet access for ICX itself. ICX can access the router on the directly connected interface. It can also access remote VPN subnets without issues. But it cannot access the internet.
Understandable, but the problem relies on what I and @fohdeesha stated before: for its comunication the switch itself will choose the first address configured, and if it resides on an unknown network on the ER side, it can't route the traffic back for the response... you can define a IP range (that belongs to the OSPF area 0) for the loopback0, so the ER will be aware of the switch address and it can correctly route the traffic back...
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
I don't need internet for those two subnets - they are for storage connections inside my lan, thus I did not include them into OSPF routing. Router does not need to know about them at all.
The only problem is internet access for ICX itself. ICX can access the router on the directly connected interface. It can also access remote VPN subnets without issues. But it cannot access the internet.

Code:
SSH@ICX6610-48P#ping 8.8.8.8
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#ping 192.168.29.254 (EDGE ROUTER)
Sending 1, 16-byte ICMP Echo to 192.168.29.254, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.29.254  : bytes=16 time<1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.
SSH@ICX6610-48P#ping 192.168.7.1 (REMOVE SUBNET)
Sending 1, 16-byte ICMP Echo to 192.168.7.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.7.1     : bytes=16 time=14ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=14/14/14 ms.
SSH@ICX6610-48P#ping 192.168.23.1 (REMOTE SUBNET)
Sending 1, 16-byte ICMP Echo to 192.168.23.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.23.1    : bytes=16 time=3ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=3/3/3 ms.
You aren't understanding the issue, so I'll try and explain it a third and final time before moving on and letting you try and figure it out on your own. Your ICX does have internet access, as evidenced by the fact every IP address on the ICX that also has a proper return route on the edgerouter can ping out to 8.8.8.8. Which leaves me to assume what you actually want is not "the ICX needs to have internet access (because it already does, on like 6 of it's 8 addresses, but you're not happy about it) but you actually mean "I want the ICX to have internet access from EVERY IP it owns". For this, I've provided the commands to solve this on your edgerouter twice now.

The ping command (and a lot of other commands, when not supplied with a specific source interface to use) will default to just using the lowest owned IP as a traffic source. On your switch that means 10.0.0.1, which, again, has no return route on the edgerouter, so there's no way for internet traffic to get back to the ICX. To break it down even simpler, which seems to be required here, this means the command you keep running to verify "internet access", "ping 8.8.8.8", is the equivalent of running ping 8.8.8.8 source 10.0.0.1" - and as established 20 times now, the edgerouter has no return route for this ICX address, and I've provided the commands to fix it. If you REALLY don't want this address having internet access, then learn to specify the ICX source address in commands requiring internet access so it doesn't default to using 10.0.0.1, and choose an ICX source address that does have a return route. example: "ping 8.8.8.8 source 192.168.0.11" - like magic, your ICX will have internet access. Same with other protocols on the switch, for instance NTP can be told which source address to use with "source-interface ve 10"

This is not a NAT issue, it's a routing issue, your edge router does not have return routes for a handful of your ICX addresses. The addresses that do have return routes do have internet access, the addresses that do not have return routes do not have internet access. I'm not sure how else to make this any clearer besides painting pictures
 

dashpuppy

Member
Dec 16, 2018
48
21
8
You aren't understanding the issue, so I'll try and explain it a third and final time before moving on and letting you try and figure it out on your own. Your ICX does have internet access, as evidenced by the fact every IP address on the ICX that also has a proper return route on the edgerouter can ping out to 8.8.8.8. Which leaves me to assume what you actually want is not "the ICX needs to have internet access (because it already does, on like 6 of it's 8 addresses, but you're not happy about it) but you actually mean "I want the ICX to have internet access from EVERY IP it owns". For this, I've provided the commands to solve this on your edgerouter twice now.

The ping command (and a lot of other commands, when not supplied with a specific source interface to use) will default to just using the lowest owned IP as a traffic source. On your switch that means 10.0.0.1, which, again, has no return route on the edgerouter, so there's no way for internet traffic to get back to the ICX. To break it down even simpler, which seems to be required here, this means the command you keep running to verify "internet access", "ping 8.8.8.8", is the equivalent of running ping 8.8.8.8 source 10.0.0.1" - and as established 20 times now, the edgerouter has no return route for this ICX address, and I've provided the commands to fix it. If you REALLY don't want this address having internet access, then learn to specify the ICX source address in commands requiring internet access so it doesn't default to using 10.0.0.1, and choose an ICX source address that does have a return route. example: "ping 8.8.8.8 source 192.168.0.11" - like magic, your ICX will have internet access. Same with other protocols on the switch, for instance NTP can be told which source address to use with "source-interface ve 10"

This is not a NAT issue, it's a routing issue, your edge router does not have return routes for a handful of your ICX addresses. The addresses that do have return routes do have internet access, the addresses that do not have return routes do not have internet access. I'm not sure how else to make this any clearer besides painting pictures

Calm down man :p take a deep breath :) we appreciate your help.. Need a coffee ?
 

dashpuppy

Member
Dec 16, 2018
48
21
8
I guess I just have this weird tick where the first four times I have to explain the same thing it's no problem, but the fifth and above start to get frustrating. Weird, I know

It's ok man ! Sometimes it just needed to be worded differently OR example given :)

Example The car is red, with 4 tires but one tire is flat.. we know it is red and has flat tire BUT the other person doesn't see it that way.

Hey Your red car has a flat tire.. :)

All to do with perception :p
 

sash

Member
Nov 22, 2019
44
8
8
...If you REALLY don't want this address having internet access, then learn to specify the ICX source address in commands requiring internet access so it doesn't default to using 10.0.0.1, and choose an ICX source address that does have a return route. example: "ping 8.8.8.8 source 192.168.0.11" - like magic, your ICX will have internet access. Same with other protocols on the switch, for instance NTP can be told which source address to use with "source-interface ve 10"

This is not a NAT issue, it's a routing issue, your edge router does not have return routes for a handful of your ICX addresses. The addresses that do have return routes do have internet access, the addresses that do not have return routes do not have internet access. I'm not sure how else to make this any clearer besides painting pictures
Appreciate your input. I will have to reconfigure NTP to use source address of virtual interface that is configured with OSPF routing then. That seems to be the solution in my case as I do not want 10.0.0.x subnet to have internet access. It is still strange to me why ICX would default to using lowest subnet configure for its own traffic, i.e. NTP and not use directly connected IP 192.168.29.1 as source.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Appreciate your input. I will have to reconfigure NTP to use source address of virtual interface that is configured with OSPF routing then. That seems to be the solution in my case as I do not want 10.0.0.x subnet to have internet access. It is still strange to me why ICX would default to using lowest subnet configure for its own traffic, i.e. NTP and not use directly connected IP 192.168.29.1 as source.
Routers/l3 switches typically get configured with a lot of IP addresses in practice, so it's been commom across vendors for some time now that when left to guess, it picks the lowest IP address to use, so there is at least some convention to go by (versus just picking source addresses at random). The switch doesn't really have the same understanding of "direct connected", in a lot of other configurations this would mean something totally different. This is all why most commands allow you to override the fallback source IP address and specify your own
 

ajax3712

New Member
May 22, 2018
7
2
3
Currently have a ICX-7250-48p (much thanks to @fohdeesha for the info & guides!). Still settings things up and still learning, but I am excited to play around. I have what may be a silly question... Is it possible or even advisable to do mixed stacking with the 7250 with other models (e.g. 7150)?

When I searched the forums for "mixed stack", I only saw the 6610/6450 come up as results. Brocade docs only discuss stacking with other 7250s...
 
  • Like
Reactions: fohdeesha