Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

kpfleming

Active Member
Dec 28, 2021
433
222
43
Pelham NY USA
firewall/router2.5 G copper linkICX 715010G fiber linkicx 7250
172.17.1.1GW: 172.17.1.1GW 172.17.1.1
172.17.10.1172.17.1.2172.17.1.3
172.17.20.1172.17.20.2172.17.20.3
The GW addresses for the .10 subnet appear to be incorrect, although that's not what you asked about :) They have to be addresses in the .10 subnet.

When setting the IP address for the ILO/iDRAC devices, I can assign 172.17.20.1 as the gateway and it works fine, however for the ESXi management configuration of the servers connected to the SFP+ ports I must set the gateway to 172.17.20.3 (e.g. the switch address on the management network, not the firewall’s address). If I use the test management network feature of ESXi, it indicates that 172.17.20.1 pings fail.

It very much appears that the SFP+ trunk/stack ports act differently than the 1G ports in this respect. If I switch the ESXi host to 1G and plug in to a 1G port instead of the SFP+ port I can use the 172.17.20.1 address as the gateway.
This is quite odd; the ports should behave the same if configured the same way. Without reviewing your configuration it's going to take a lot of questions, but... starting with are you running the layer2 (S) or layer3 (R) firmware on the switches? Are the links between the firewall/switch/switch devices 'trunk' links with all three VLANs on them?
 

rory

New Member
May 28, 2021
7
2
3
The GW addresses for the .10 subnet appear to be incorrect, although that's not what you asked about :) They have to be addresses in the .10 subnet.



This is quite odd; the ports should behave the same if configured the same way. Without reviewing your configuration it's going to take a lot of questions, but... starting with are you running the layer2 (S) or layer3 (R) firmware on the switches? Are the links between the firewall/switch/switch devices 'trunk' links with all three VLANs on them?
Fair enough comment about configurations. I guess my diagram was misunderstood; there is only a single GW address defined, the others are addresses on the virtual interfaces. That said, here are the respective config files with minor redactions. Yes, the fiber link between the switches is trunked and carries all three vlans; as I said the setup is working with the exception of my not understanding what is different about the SFP+ ports on the 7250.

Office switch config file:
Code:
SSH@office-1>sh run
Current configuration:
!
ver 08.0.95pT213
!
stack unit 1
  module 1 icx7150-24p-poe-port-management-module
  module 2 icx7150-2-copper-port-2g-module
  module 3 icx7150-4-sfp-plus-port-40g-module
  stack-port 1/3/1
  stack-port 1/3/3
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
no untagged ethe 1/1/21 ethe 1/1/23
router-interface ve 1
!
vlan 10 name IoT by port
tagged ethe 1/1/21 to 1/1/22 ethe 1/2/2 ethe 1/3/4
!
vlan 20 name Management by port
tagged ethe 1/1/3 ethe 1/1/5 ethe 1/1/22 to 1/1/24 ethe 1/2/2 ethe 1/3/3 to 1/3/4
router-interface ve 20
!
!
!
!
!
!
!
!
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
chassis fanless all
enable aaa console
hostname office-1
ip dhcp-client disable                                         
ip dns server-address 172.17.1.1
ip route 0.0.0.0/0 172.17.1.1
!
logging host 172.17.20.10  udp-port 6514
no telnet server
username <removed>
username <removed>
!
!
snmp-server community ..... ro
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT-08
!
!
ntp
disable serve
server time.apple.com
server time1.google.com
!
!                                                               
web access-group 1
ssh access-group 1
!
manager disable
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!

<various port/machine name info removed>

interface ve 1
ip address 172.17.1.2 255.255.255.0
!
interface ve 20
ip address 172.17.20.2 255.255.255.0
!
!
!
ip access-list standard 1
sequence 10 permit 172.17.1.0 0.0.0.31
sequence 20 permit 172.17.20.0 0.0.0.255
!
!
!
!
!
!
!
!
ip ssh  idle-time 0                                             
!
!
!
!
!
end
SSH@office-1>
Rack switch config:
Code:
SSH@rack-1>sh run
Current configuration:
!
ver 08.0.95pT213
!
stack unit 1
  module 1 icx7250-24p-poe-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
  stack-port 1/2/1
  stack-port 1/2/3
!
!
!
lag TrueNAS dynamic id 256
lacp-timeout long
ports ethe 1/2/1 to 1/2/2
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 name IoT by port
tagged ethe 1/1/23 to 1/1/24 ethe 1/2/7 to 1/2/8
!                                                                
vlan 20 name Management by port
tagged ethe 1/1/1 to 1/1/12 ethe 1/2/3 to 1/2/8 lag 256
router-interface ve 20
!
!
!
!
!
!
!
!
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname rack-1                                                  
ip dhcp-client disable
ip dns server-address 172.17.1.1
ip route 0.0.0.0/0 172.17.1.1
!
logging host 172.17.20.10  udp-port 6514
no telnet server
username <removed>
username <removed>
!
!
snmp-server community ..... ro
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT-08
!
!
ntp
disable serve
server time.apple.com
server time1.google.com
!                                                                
!
web access-group 1
ssh access-group 1
!
manager disable
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/13
no inline power
!
interface ethernet 1/2/3
no optical-monitor
!                                                                
interface ethernet 1/2/4
no optical-monitor
!
interface ethernet 1/2/5
no optical-monitor
!
interface ethernet 1/2/6
no optical-monitor
!
interface ethernet 1/2/7
no optical-monitor
!
interface ethernet 1/2/8
no optical-monitor
!
interface ve 1
ip address 172.17.1.3 255.255.255.0
!
interface ve 20
ip address 172.17.20.3 255.255.255.0
!
!
!                                                                
ip access-list standard 1
sequence 10 permit 172.17.1.0 0.0.0.31
sequence 20 permit 172.17.20.0 0.0.0.255
!
!
!
!
!
!
!
!
ip ssh  idle-time 0
!
!
!
!
!
end
SSH@rack-1>
 
Last edited:

kpfleming

Active Member
Dec 28, 2021
433
222
43
Pelham NY USA
Fair enough comment about configurations. I guess my diagram was misunderstood; there is only a single GW address defined, the others are addresses on the virtual interfaces. That said, here are the respective config files with minor redactions. Yes, the fiber link between the switches is trunked and carries all three vlans; as I said the setup is working with the exception of my not understanding what is different about the SFP+ ports on the 7250.
On both switches you have the default stacking configuration in place, which assigns two of the SFP+ ports for stacking. You've also got one of those 'stack ports' on each switch in VLAN 20, so that will not give you the results you want (I'm surprised you can get any traffic to pass through those ports at all).

I suggest at a minimum using `no stack-port` to remove the ports you want to use from the stacking configuration; you may need to do more that that, I can't remember for sure.
 
  • Like
Reactions: rory

rory

New Member
May 28, 2021
7
2
3
Thank you!

Removing the default stack ports solved this. As implied by the name I did not configure or add the stack ports, they were enabled by default. I did not think to remove them since with the exception of having to use a different default gateway on the SFP+ ports for VLAN 20, everything else was working perfectly.

It is somewhat interesting that the true inter-switch trunk was NOT one of the stack-ports...

OCD satisfied ;-)

-Rory
 
Last edited:
  • Like
Reactions: kpfleming

jmdomini

New Member
Jan 30, 2024
16
0
1
Trying to setup a VLAN between my two Brocade 6450's and they are kicking my a$$. Can anyone explain why the below doesn't work (can't pass any data across the VLAN)? In the Cisco world I would do "switchport trunk allowed vlan all". However there is no such think with Brocade. All I want to do is pass VLAN2 across than same fibre that is carrying the switches native VLAN1 and split it out to port 1/1/13 on the connected workstation switch. It shouldn't be this hard!

Core Switch
vlan 2 by port
tagged ethe 1/1/23 ethe 1/2/1 ethe 1/2/3
spanning-tree

!
interface ethernet 1/2/1 <--To PFSense Router
dual-mode
interface ethernet 1/2/3 <--To Workstation Switch
dual-mode


Workstation Switch
vlan 2 name Wirguard-VLAN by port
tagged ethe 1/1/13 ethe 1/2/4
spanning-tree

!
interface ethernet 1/2/4 -->Uplink from core switch
dual-mode
 

cyinite

New Member
Jun 28, 2024
8
5
3
Trying to setup a VLAN between my two Brocade 6450's and they are kicking my a$$. Can anyone explain why the below doesn't work (can't pass any data across the VLAN)? In the Cisco world I would do "switchport trunk allowed vlan all". However there is no such think with Brocade. All I want to do is pass VLAN2 across than same fibre that is carrying the switches native VLAN1 and split it out to port 1/1/13 on the connected workstation switch. It shouldn't be this hard!

Core Switch
vlan 2 by port
tagged ethe 1/1/23 ethe 1/2/1 ethe 1/2/3
spanning-tree

!
interface ethernet 1/2/1 <--To PFSense Router
dual-mode
interface ethernet 1/2/3 <--To Workstation Switch
dual-mode


Workstation Switch
vlan 2 name Wirguard-VLAN by port
tagged ethe 1/1/13 ethe 1/2/4
spanning-tree

!
interface ethernet 1/2/4 -->Uplink from core switch
dual-mode
I assume that your PFsense box is the one routing between VLANs? Does your router and core switch know the routes to take to get to VLAN 1 and VLAN 2 on the workstation switch? I've had issues in the past where my router didn't know where the rest of VLAN 1 was...

And to double check, dual-mode 1 was the command used on the eth interfaces tagged with VLAN 2?
 

jmdomini

New Member
Jan 30, 2024
16
0
1
I assume that your PFsense box is the one routing between VLANs? Does your router and core switch know the routes to take to get to VLAN 1 and VLAN 2 on the workstation switch? I've had issues in the past where my router didn't know where the rest of VLAN 1 was...

And to double check, dual-mode 1 was the command used on the eth interfaces tagged with VLAN 2?
Hmm, I guess I'm confused then. It's been a while, but I don't recall having to do any routing to pass VLAN's between switches. I used to have some ancient Cisco switches and I seem to think that I could have say VLAN10 on both switches and assign ports on both. Do switch trunk allowed vlan all on the connection between them and devices connected to ports associated to the same VLAN at both ends could communicate with each other.

Sounds like you are saying it doesn't work that with Brocade switches and I need to somehow tell switch A that VLAN 2 exist on x port on switch B as well. Where as I'm assuming that dual-mode essential works like "switchport trunk allowed vlan all" and I'm expecting VLAN 2 to automagically carry over to switch B because I allowed it to pass on the "trunk" using this command.
 

kpfleming

Active Member
Dec 28, 2021
433
222
43
Pelham NY USA
Sounds like you are saying it doesn't work that with Brocade switches and I need to somehow tell switch A that VLAN 2 exist on x port on switch B as well.
From what I can tell the ICX devices don't have a 'trunk all VLANs' mode like other switches do; every VLAN which should be carried over a port has to be configured on that port. 'dual mode' only allows simultaneous usage of tagged and untagged traffic on the port, it doesn't indicate which VLANs should be carried on the port in tagged mode; that must be done by adding the port to those VLANs using 'tagged eth 1/1/10' commands, or something similar.
 
  • Like
Reactions: cyinite and BoGs

Andydude

New Member
Oct 6, 2023
12
7
3
SNMP shows Errors in: 0.029% but port statistics on the interface show everything is fine?

Hi Guys, long shot but thought I'd see if anyone has seen this before. After seeing this thread, I bought a 7250 and 6450 and they've been running great for over a year with two 10Gb links between the two.

Recently I'm seeing warnings from my SNMP monitoring system CheckMK just on just the 6450 that's showing things like this and then quickly resolve themselves:
[FibreUplink1], (up), MAC: 74:8E:F8:D7:D0:40, Speed: 10 GBit/s, In: 127 kB/s (0.01%), Out: 27.6 kB/s (<0.01%), Errors in: 0.035% (warn/crit at 0.01%/0.1%)
[FibreUplink2], (up), MAC: 74:8E:F8:D7:D0:40, Speed: 10 GBit/s, In: 67.3 kB/s (0.01%), Out: 40.7 kB/s (0.01%), Errors in: 0.051% (warn/crit at 0.01%/0.1%)(!)

However, if I go and check the port stats on the interfaces they are showing that there are 0 Errors In? Not sure if it's an SNMP issue, false positive or something wrong with CheckMK. Have raised it on the forum there as well but no replies as of yet.

Anyone got any insights? I've done all the normal things like swap the cables, restart the switch, power cycle the switch, replace transceivers etc.

1734452679912.png
1734452776054.png
 

jmdomini

New Member
Jan 30, 2024
16
0
1
From what I can tell the ICX devices don't have a 'trunk all VLANs' mode like other switches do; every VLAN which should be carried over a port has to be configured on that port. 'dual mode' only allows simultaneous usage of tagged and untagged traffic on the port, it doesn't indicate which VLANs should be carried on the port in tagged mode; that must be done by adding the port to those VLANs using 'tagged eth 1/1/10' commands, or something similar.
I think what I need is an example, or site explaining how to setup a vlan that spans two switches. There's plenty of example of setting up a vlan within the confines of a single switch, and I can get that working fine. But getting a vlan to span switches is nigh impossible and there seems to be zero documentation or examples out there on how to do this.
 

kapone

Well-Known Member
May 23, 2015
1,285
738
113
I think what I need is an example, or site explaining how to setup a vlan that spans two switches. There's plenty of example of setting up a vlan within the confines of a single switch, and I can get that working fine. But getting a vlan to span switches is nigh impossible and there seems to be zero documentation or examples out there on how to do this.
Something like...

Switch 1
conf t
vlan 1,2,3,4,5..... (All VLANs that need to flow, in one go)
tag eth x/x/x (The single port that's connected to switch 2)

Switch 2 (assuming it's a Brocade as well)
conf t
vlan 1,2,3,4,5..... (All VLANs that need to flow, in one go)
tag eth y/y/y (The single port that's connected to switch 1)

Would that not work?
 

rory

New Member
May 28, 2021
7
2
3
I posted the full configs for my two switches 8 posts above this. I have two vlans (10 and 20) that span my two switches. The "trunk" connecting the two switches is on port 1/3/4 in the office switch and port 1/2/8 in the rack switch.

The only real difference for you is that I am running a newer version of the switch firmware that does not require the dual-mode command, which you will need to add to the ports connecting the switches (as well as any other port which needs to carry both tagged and untagged packets).
 

kpfleming

Active Member
Dec 28, 2021
433
222
43
Pelham NY USA
The only real difference for you is that I am running a newer version of the switch firmware that does not require the dual-mode command
Same here, which is why I didn't immediately offer a proposed configuration... I don't know what is different between those configuration styles.
 

jmdomini

New Member
Jan 30, 2024
16
0
1
Thank you both for trying to help. Seems I have bigger problems as I came back to a dead switch. Appears the power supply toasted on my core switch (burning smell). So I've had to put back my old Cisco switch for now. I'll explore my options for replacement after the holidays.
 

rory

New Member
May 28, 2021
7
2
3
I used to tun a 6610 in my rack which required the "dual mode". Great switch but I did not need all 48 ports and wanted to cut down on the power/noise.
 

SJENSEN721

New Member
Aug 25, 2024
4
1
3
You can absolutely use a single regular qsfp cable to connect the breakout port to another switch (assuming the other switches port is configured for breakout / 4x 10gbe). You can even connect a breakout port with a qsfp cable to a non breakout port on something, but only one lane will link up so it will run at 10gbps

Both 40gbe and 4x 10gbe breakout qsfp+ are electrically identical, it's 4x 10gbps lanes. Difference is in how the ASIC/PHY decides to split them (or mux them)
Hey @fohdeesha and @NablaSquaredG,

You two are champions in the ICX 6610 space and I'm running into a wall.

I have 3 6610s, stacked. I want a ring topology for the three of them.

I have an OPNSense router connected to 1/2/1 at 40G (I'd like to keep this here because my card doesn't mux). I have a Cisco N3K connected to 3/2/7-3/2/10 (it muxes to 40G). I have a Mellanox SX6036 connected to 3/2/1 at 40G (I'd like to keep this connection here because I don't know how to create a setting on the SX6036 to mux 4x10G lanes).

My currently linear topology is: 1/2/6 -> 2/2/6, 2/2/1 -> 3/2/1... I would like to connect 3/2/2-3/2/5 -> 1/2/2-1/2/5. My cable is: https://www.amazon.com/gp/product/B01N4GKPQ8/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1

I'm a bit new to networking, so I'm pretty ignorant. Long story short, it seems no matter what I do, I can't get the stack to move to a ring topology when using the connection 3/2/2-3/2/5 -> 1/2/2-1/2/5.

I'd frankly prefer to use all of the breakout ports for my ring topology and save the pure 40G ports for other things, but I couldn't get the stack to configure on the breakout ports. Do either of you know how to help my sad/ignorant self?
 
Last edited:

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,691
1,123
113
I'd frankly prefer to use all of the breakout ports for my ring topology and save the pure 40G ports for other things, but I couldn't get the stack to configure on the breakout ports. Do either of you know how to help my sad/ignorant self?
I remember having a similar issue a while back.

According to the manual, you can use a "ring stack topologies with partial cabling" - however it seems like one 40G port for stacking is always required. I'm also not sure whether you deconfigure the stack-trunk (which is the thing that allows using the ports as data ports) and use partial cabling at the same time.

If I remember correctly, I used 40G for stacking and configured a 4x10G trunk to the SX6036 with 4x10G link aggregation on the SX6036. It's not that difficult
 

kapone

Well-Known Member
May 23, 2015
1,285
738
113
After the...longest...time, I think it's time to upgrade. :) Ran out of all 10g/40g ports on my 6610...need more (adding some more production capacity).

Sanity check - Arista 7050QX-32S still the most viable option? (has more than enough 10g/40g ports. Would have loved some 2.5/25gb ports as well, but it's not critical)

Edit: Anybody interested in a modded 6610 with 3x 120mm fans on top (hacked up top cover) and NO PSU fans (PWM trickery...)? Been running stable for years...
 

autoturk

Active Member
Sep 1, 2022
236
200
43
just a heads up, the icx7150-24p/48p (and non-P variants!) can be had for <$200 on ebay these days, and these are arguably a contender for the "go-to" homelab switch. You can run it fanless (no need for any mods, just a simple CLI command) if you are okay with 150 watts of PoE output. They also sip power (15ish watts for non-P and 30-ish for P)