Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[AranoX]

Hi guys!

Got my 6450 and i'm pretty new to the Brocade stuff, basic config was no problem till now, but now im a little stuck...

I'm trying to connect my 5G router (Zyxel NR7302, ) to eth1 and pass it trough to eth2 (currently connect to LAN1 on FRITZbox 7530, will be a opnSense Box shortly).
It seems that the whole installation does not like it when the switch inbetween... but i dont get why.

Can somebody maybe guide me in the right direction on how to configurate eth1 and eth2 so that WAN Port (LAN1) on FRITZbox is working again?

 

[samalewis]

Hi all,
Referencing post #4, is it possible to connect two breakout ports directly with Brocade 57-1000267-01 Xbr-000232 4 X 16gb QSFP MPO optics? I would like to replace my current dynamic lag (from Sw1 1/3/5-8 to sw2 1/2/7-10) that uses a breakout cable and go directly from sw1 1/2/7-10 to sw2 1/2/7-10 using the MPO 4x10 optics on both sides connected with a single MPO cable.
My use case requires all four 40gbe ports (two per switch) to be usable so this is my current working solution, but it is clunky and more cabling than I would like to have to manage. I have tried searching/sifting the thread and haven't found a direct answer. Thanks!

 

[duplexsystem]

I have bought some ptm7950 from linus for my 6610, and I am going to try and replace the existing thermal paste with this, I went with ptm7950 over paste because it seems like it will last longer because I really don't want to have to open this up more than once. I am hoping to improve thermals thereby improving fan noise. But even if thermals are improved with no fan noise improvement would be good as well. I will post before and after thermals once I do this. If anyone else has done something similar let me know how it went!

 

[rory]

Help me understand what is going on in my homelab setup:

172.17.1.XX vlan1 trusted machines
172.17.10.XX vlan10 untrusted machines (IoT & guest)
172.17.20.XX vlan20 management (ILO, ESXi hosts, etc)

ICX7150 is in my office with desktops, printers, scanners
ICX7250 is in my garage with half rack and servers

firewall/router	2.5 G copper link	ICX 7150	10G fiber link	icx 7250
172.17.1.1		GW: 172.17.1.1		GW 172.17.1.1
172.17.10.1		172.17.1.2		172.17.1.3
172.17.20.1		172.17.20.2		172.17.20.3

Everything works fine… but I have one annoyance I don’t understand.

In my half rack I have the main network connections for my servers hooked up via DAC to the 10G SFP+ ports, but their ILO/iDRAC connections are hooked up to standard 1G ports.

The link between the two switches is a 10G fiber connection utilizing a SFP+ port on each switch.

When setting the IP address for the ILO/iDRAC devices, I can assign 172.17.20.1 as the gateway and it works fine, however for the ESXi management configuration of the servers connected to the SFP+ ports I must set the gateway to 172.17.20.3 (e.g. the switch address on the management network, not the firewall’s address). If I use the test management network feature of ESXi, it indicates that 172.17.20.1 pings fail when the gateway is set to 172.17.20.1. If I set the gateway to 172.17.20.3, the test management network pings of 172.17.20.1 succeed.

It very much appears that the SFP+ trunk/stack ports act differently than the 1G ports in this respect. If I switch the ESXi host to 1G and plug in to a 1G port instead of the SFP+ port I can use the 172.17.20.1 address as the gateway. It acts as if the switch is not forwarding ARP requests for the VLAN between SFP+ ports.

What is going on, and to satisfy my OCD nature, is there a way for me to not have to specify 172.17.20.3 as the gateway for servers connected via SFP+ ports?

Thanks in advance!

 

[56ip1234]

Does the ICX7150-48ZP have any licensing issues with the SFP+ ports or can I enable the licenses like I have with my ICX7150-48P? I see sellers listing different models with different numbers of SFP+ active.

 

[rory]

Does the ICX7150-48ZP have any licensing issues with the SFP+ ports or can I enable the licenses like I have with my ICX7150-48P? I see sellers listing different models with different numbers of SFP+ active.

I am pretty sure that all the 7XXX series have gone to the honor based licensing. Although I do not own the 7150-48ZP, I own several 7150-24P units that were advertised with different numbers of SFP+ ports. The only difference was the installed license, and I was able to simply enable the additional ports.

 

[56ip1234]

I am pretty sure that all the 7XXX series have gone to the honor based licensing. Although I do not own the 7150-48ZP, I own several 7150-24P units that were advertised with different numbers of SFP+ ports. The only difference was the installed license, and I was able to simply enable the additional ports.

That's my understanding but given the price tag and the main reason to upgrade is the additional SFP+ I hope someone can definitively confirm. The 2.5 Gbps ports are a nice plus.

 

[kpfleming]

firewall/router	2.5 G copper link	ICX 7150	10G fiber link	icx 7250
172.17.1.1		GW: 172.17.1.1		GW 172.17.1.1
172.17.10.1		172.17.1.2		172.17.1.3
172.17.20.1		172.17.20.2		172.17.20.3

The GW addresses for the .10 subnet appear to be incorrect, although that's not what you asked about They have to be addresses in the .10 subnet.

When setting the IP address for the ILO/iDRAC devices, I can assign 172.17.20.1 as the gateway and it works fine, however for the ESXi management configuration of the servers connected to the SFP+ ports I must set the gateway to 172.17.20.3 (e.g. the switch address on the management network, not the firewall’s address). If I use the test management network feature of ESXi, it indicates that 172.17.20.1 pings fail.

It very much appears that the SFP+ trunk/stack ports act differently than the 1G ports in this respect. If I switch the ESXi host to 1G and plug in to a 1G port instead of the SFP+ port I can use the 172.17.20.1 address as the gateway.

This is quite odd; the ports should behave the same if configured the same way. Without reviewing your configuration it's going to take a lot of questions, but... starting with are you running the layer2 (S) or layer3 (R) firmware on the switches? Are the links between the firewall/switch/switch devices 'trunk' links with all three VLANs on them?

 

[rory]

The GW addresses for the .10 subnet appear to be incorrect, although that's not what you asked about They have to be addresses in the .10 subnet.



This is quite odd; the ports should behave the same if configured the same way. Without reviewing your configuration it's going to take a lot of questions, but... starting with are you running the layer2 (S) or layer3 (R) firmware on the switches? Are the links between the firewall/switch/switch devices 'trunk' links with all three VLANs on them?

Fair enough comment about configurations. I guess my diagram was misunderstood; there is only a single GW address defined, the others are addresses on the virtual interfaces. That said, here are the respective config files with minor redactions. Yes, the fiber link between the switches is trunked and carries all three vlans; as I said the setup is working with the exception of my not understanding what is different about the SFP+ ports on the 7250.

Office switch config file:

SSH@office-1>sh run
Current configuration:
!
ver 08.0.95pT213
!
stack unit 1
  module 1 icx7150-24p-poe-port-management-module
  module 2 icx7150-2-copper-port-2g-module
  module 3 icx7150-4-sfp-plus-port-40g-module
  stack-port 1/3/1
  stack-port 1/3/3
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
no untagged ethe 1/1/21 ethe 1/1/23
router-interface ve 1
!
vlan 10 name IoT by port
tagged ethe 1/1/21 to 1/1/22 ethe 1/2/2 ethe 1/3/4
!
vlan 20 name Management by port
tagged ethe 1/1/3 ethe 1/1/5 ethe 1/1/22 to 1/1/24 ethe 1/2/2 ethe 1/3/3 to 1/3/4
router-interface ve 20
!
!
!
!
!
!
!
!
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
chassis fanless all
enable aaa console
hostname office-1
ip dhcp-client disable                                         
ip dns server-address 172.17.1.1
ip route 0.0.0.0/0 172.17.1.1
!
logging host 172.17.20.10  udp-port 6514
no telnet server
username <removed>
username <removed>
!
!
snmp-server community ..... ro
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT-08
!
!
ntp
disable serve
server time.apple.com
server time1.google.com
!
!                                                               
web access-group 1
ssh access-group 1
!
manager disable
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!

<various port/machine name info removed>

interface ve 1
ip address 172.17.1.2 255.255.255.0
!
interface ve 20
ip address 172.17.20.2 255.255.255.0
!
!
!
ip access-list standard 1
sequence 10 permit 172.17.1.0 0.0.0.31
sequence 20 permit 172.17.20.0 0.0.0.255
!
!
!
!
!
!
!
!
ip ssh  idle-time 0                                             
!
!
!
!
!
end
SSH@office-1>

Rack switch config:

SSH@rack-1>sh run
Current configuration:
!
ver 08.0.95pT213
!
stack unit 1
  module 1 icx7250-24p-poe-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
  stack-port 1/2/1
  stack-port 1/2/3
!
!
!
lag TrueNAS dynamic id 256
lacp-timeout long
ports ethe 1/2/1 to 1/2/2
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 name IoT by port
tagged ethe 1/1/23 to 1/1/24 ethe 1/2/7 to 1/2/8
!                                                                
vlan 20 name Management by port
tagged ethe 1/1/1 to 1/1/12 ethe 1/2/3 to 1/2/8 lag 256
router-interface ve 20
!
!
!
!
!
!
!
!
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname rack-1                                                  
ip dhcp-client disable
ip dns server-address 172.17.1.1
ip route 0.0.0.0/0 172.17.1.1
!
logging host 172.17.20.10  udp-port 6514
no telnet server
username <removed>
username <removed>
!
!
snmp-server community ..... ro
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT-08
!
!
ntp
disable serve
server time.apple.com
server time1.google.com
!                                                                
!
web access-group 1
ssh access-group 1
!
manager disable
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/13
no inline power
!
interface ethernet 1/2/3
no optical-monitor
!                                                                
interface ethernet 1/2/4
no optical-monitor
!
interface ethernet 1/2/5
no optical-monitor
!
interface ethernet 1/2/6
no optical-monitor
!
interface ethernet 1/2/7
no optical-monitor
!
interface ethernet 1/2/8
no optical-monitor
!
interface ve 1
ip address 172.17.1.3 255.255.255.0
!
interface ve 20
ip address 172.17.20.3 255.255.255.0
!
!
!                                                                
ip access-list standard 1
sequence 10 permit 172.17.1.0 0.0.0.31
sequence 20 permit 172.17.20.0 0.0.0.255
!
!
!
!
!
!
!
!
ip ssh  idle-time 0
!
!
!
!
!
end
SSH@rack-1>

 

[kpfleming]

Fair enough comment about configurations. I guess my diagram was misunderstood; there is only a single GW address defined, the others are addresses on the virtual interfaces. That said, here are the respective config files with minor redactions. Yes, the fiber link between the switches is trunked and carries all three vlans; as I said the setup is working with the exception of my not understanding what is different about the SFP+ ports on the 7250.

On both switches you have the default stacking configuration in place, which assigns two of the SFP+ ports for stacking. You've also got one of those 'stack ports' on each switch in VLAN 20, so that will not give you the results you want (I'm surprised you can get any traffic to pass through those ports at all).

I suggest at a minimum using `no stack-port` to remove the ports you want to use from the stacking configuration; you may need to do more that that, I can't remember for sure.

 

[rory]

Thank you!

Removing the default stack ports solved this. As implied by the name I did not configure or add the stack ports, they were enabled by default. I did not think to remove them since with the exception of having to use a different default gateway on the SFP+ ports for VLAN 20, everything else was working perfectly.

It is somewhat interesting that the true inter-switch trunk was NOT one of the stack-ports...

OCD satisfied ;-)

-Rory

 

[jmdomini]

Trying to setup a VLAN between my two Brocade 6450's and they are kicking my a$$. Can anyone explain why the below doesn't work (can't pass any data across the VLAN)? In the Cisco world I would do "switchport trunk allowed vlan all". However there is no such think with Brocade. All I want to do is pass VLAN2 across than same fibre that is carrying the switches native VLAN1 and split it out to port 1/1/13 on the connected workstation switch. It shouldn't be this hard!

Core Switch
vlan 2 by port
tagged ethe 1/1/23 ethe 1/2/1 ethe 1/2/3
spanning-tree

!
interface ethernet 1/2/1 <--To PFSense Router
dual-mode
interface ethernet 1/2/3 <--To Workstation Switch
dual-mode


Workstation Switch
vlan 2 name Wirguard-VLAN by port
tagged ethe 1/1/13 ethe 1/2/4
spanning-tree

!
interface ethernet 1/2/4 -->Uplink from core switch
dual-mode

 

[cyinite]

Trying to setup a VLAN between my two Brocade 6450's and they are kicking my a$$. Can anyone explain why the below doesn't work (can't pass any data across the VLAN)? In the Cisco world I would do "switchport trunk allowed vlan all". However there is no such think with Brocade. All I want to do is pass VLAN2 across than same fibre that is carrying the switches native VLAN1 and split it out to port 1/1/13 on the connected workstation switch. It shouldn't be this hard!

Core Switch
vlan 2 by port
tagged ethe 1/1/23 ethe 1/2/1 ethe 1/2/3
spanning-tree

!
interface ethernet 1/2/1 <--To PFSense Router
dual-mode
interface ethernet 1/2/3 <--To Workstation Switch
dual-mode


Workstation Switch
vlan 2 name Wirguard-VLAN by port
tagged ethe 1/1/13 ethe 1/2/4
spanning-tree

!
interface ethernet 1/2/4 -->Uplink from core switch
dual-mode

I assume that your PFsense box is the one routing between VLANs? Does your router and core switch know the routes to take to get to VLAN 1 and VLAN 2 on the workstation switch? I've had issues in the past where my router didn't know where the rest of VLAN 1 was...

And to double check, dual-mode 1 was the command used on the eth interfaces tagged with VLAN 2?

 

[jmdomini]

I assume that your PFsense box is the one routing between VLANs? Does your router and core switch know the routes to take to get to VLAN 1 and VLAN 2 on the workstation switch? I've had issues in the past where my router didn't know where the rest of VLAN 1 was...

And to double check, dual-mode 1 was the command used on the eth interfaces tagged with VLAN 2?

Hmm, I guess I'm confused then. It's been a while, but I don't recall having to do any routing to pass VLAN's between switches. I used to have some ancient Cisco switches and I seem to think that I could have say VLAN10 on both switches and assign ports on both. Do switch trunk allowed vlan all on the connection between them and devices connected to ports associated to the same VLAN at both ends could communicate with each other.

Sounds like you are saying it doesn't work that with Brocade switches and I need to somehow tell switch A that VLAN 2 exist on x port on switch B as well. Where as I'm assuming that dual-mode essential works like "switchport trunk allowed vlan all" and I'm expecting VLAN 2 to automagically carry over to switch B because I allowed it to pass on the "trunk" using this command.

 

[kpfleming]

Sounds like you are saying it doesn't work that with Brocade switches and I need to somehow tell switch A that VLAN 2 exist on x port on switch B as well.

From what I can tell the ICX devices don't have a 'trunk all VLANs' mode like other switches do; every VLAN which should be carried over a port has to be configured on that port. 'dual mode' only allows simultaneous usage of tagged and untagged traffic on the port, it doesn't indicate which VLANs should be carried on the port in tagged mode; that must be done by adding the port to those VLANs using 'tagged eth 1/1/10' commands, or something similar.

 

[Andydude]

SNMP shows Errors in: 0.029% but port statistics on the interface show everything is fine?

Hi Guys, long shot but thought I'd see if anyone has seen this before. After seeing this thread, I bought a 7250 and 6450 and they've been running great for over a year with two 10Gb links between the two.

Recently I'm seeing warnings from my SNMP monitoring system CheckMK just on just the 6450 that's showing things like this and then quickly resolve themselves:
[FibreUplink1], (up), MAC: 74:8E:F870:40, Speed: 10 GBit/s, In: 127 kB/s (0.01%), Out: 27.6 kB/s (<0.01%), Errors in: 0.035% (warn/crit at 0.01%/0.1%)
[FibreUplink2], (up), MAC: 74:8E:F870:40, Speed: 10 GBit/s, In: 67.3 kB/s (0.01%), Out: 40.7 kB/s (0.01%), Errors in: 0.051% (warn/crit at 0.01%/0.1%)(!)

However, if I go and check the port stats on the interfaces they are showing that there are 0 Errors In? Not sure if it's an SNMP issue, false positive or something wrong with CheckMK. Have raised it on the forum there as well but no replies as of yet.

Anyone got any insights? I've done all the normal things like swap the cables, restart the switch, power cycle the switch, replace transceivers etc.

 

[jmdomini]

From what I can tell the ICX devices don't have a 'trunk all VLANs' mode like other switches do; every VLAN which should be carried over a port has to be configured on that port. 'dual mode' only allows simultaneous usage of tagged and untagged traffic on the port, it doesn't indicate which VLANs should be carried on the port in tagged mode; that must be done by adding the port to those VLANs using 'tagged eth 1/1/10' commands, or something similar.

I think what I need is an example, or site explaining how to setup a vlan that spans two switches. There's plenty of example of setting up a vlan within the confines of a single switch, and I can get that working fine. But getting a vlan to span switches is nigh impossible and there seems to be zero documentation or examples out there on how to do this.

 

[kapone]

I think what I need is an example, or site explaining how to setup a vlan that spans two switches. There's plenty of example of setting up a vlan within the confines of a single switch, and I can get that working fine. But getting a vlan to span switches is nigh impossible and there seems to be zero documentation or examples out there on how to do this.

Something like...

Switch 1
conf t
vlan 1,2,3,4,5..... (All VLANs that need to flow, in one go)
tag eth x/x/x (The single port that's connected to switch 2)

Switch 2 (assuming it's a Brocade as well)
conf t
vlan 1,2,3,4,5..... (All VLANs that need to flow, in one go)
tag eth y/y/y (The single port that's connected to switch 1)

Would that not work?

 

[rory]

I posted the full configs for my two switches 8 posts above this. I have two vlans (10 and 20) that span my two switches. The "trunk" connecting the two switches is on port 1/3/4 in the office switch and port 1/2/8 in the rack switch.

The only real difference for you is that I am running a newer version of the switch firmware that does not require the dual-mode command, which you will need to add to the ports connecting the switches (as well as any other port which needs to carry both tagged and untagged packets).

 

[kpfleming]

The only real difference for you is that I am running a newer version of the switch firmware that does not require the dual-mode command

Same here, which is why I didn't immediately offer a proposed configuration... I don't know what is different between those configuration styles.