Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

infoMatt

Active Member
Apr 16, 2019
157
60
28
For basic L2/L3 tasks, pretty much every vendor has the same logics and "things" to setup via the CLI.
The biggest difference in Brocade-style vs Cisco-style CLI is the definition of VLAN and interface... With the former, you attach an interface to a vlan (conf t; vlan xxx; tagged ether 1/1/3), with the latter you define VLANs on a interface (conf t; interface ethe 1/1/3; switchport tagged vlan add xxx).
HP/Aruba AFIK are Brocade-like.

Aside for the cosmetic difference and "synonyms" for the different things (dammnit Brocade, why 'disable' and not 'shutdown' a-la Cisco??), the real value of those cheap boxes is to learn networking...

It's the same for say a programming language... in Python you'll iteate in a list in a different way than in Java or C++, but you know that to make a sum of all items you'll have to iterate over all of them. ;)
 
Last edited:

jd.developer

New Member
Jan 12, 2020
8
2
3
Concerning the 40G to 10G breakout setup on the ICX6610, I can’t seem to get the 40g to 10g breakout working. I have:

  1. a 40G-QSFP-SR-INT optic (also have the infiniband optic mentioned in an earlier post)
  2. fiber: female MPO to 8x LC OM4
  3. 10 G brocade optics and intel optics
I have connected the 40G optic to a far right qsfp port (should be a breakout port according to the earlier post by fohdeesha), connected the MPO cable, and then connected the LC end to a 10g optic, but no connection is established on the 10g side. I’ve confirmed the polarity is correct / matches the arrows on the 10g optic - light is coming out of the fiber. The switch also sees the 40G optic.

I’m using intel 10g nics, and usually the moment the fiber is plugged into a 10G connection, the nic port lights up indicating activity. Switching out intel and brocade optics on the 10g side seems to not make a difference. The host is a freenas box and the logs just show the interface as down when connected through the 10g breakout.

On the switch side, the breakout ports are enabled and the ICX6610 is not configured for stacking. It is running the the latest firmware, which should have breakout support. I wasn’t able to find any documentation on brocade’s site concerning fiber based breakouts.

Does anyone have an idea on how to get the breakouts working or any documentation I could read up further?
 
Last edited:
May 1, 2020
37
8
8
I just got my ICX6450 and am working through your guide. (finished actually but I'm trying to tweak things related to it)

I was wondering about the "enable" password. I created an account and I can login via ssh with a key which makes things really easy. However, when I want to elevate to the "enable" level, I still have to enter a password. Is it possible and safe to disable the enable password? or is there a way to authenticate it with they key itself?

I would think that the enable password is something that could be safely removed and handled purely based on user access levels. If I login as root, I can "enable" with no password (since I got in with the proper credentials to begin with), but if I'm a lower user I can't enable at all. Similar to sudo privileges on Linux.
 

infoMatt

Active Member
Apr 16, 2019
157
60
28
Is it possible and safe to disable the enable password? or is there a way to authenticate it with they key itself?
It's not possible to use a key authentication during the enable phase because you're in a strictly text-only phase... and it must work even on a serial console that doesn't know anything about SSH keys.
You can however disable it; not the wiser move in my opinion, but in a home/lab environment... why not?

I would think that the enable password is something that could be safely removed and handled purely based on user access levels. If I login as root, I can "enable" with no password (since I got in with the proper credentials to begin with), but if I'm a lower user I can't enable at all. Similar to sudo privileges on Linux.
You can define different security levels:

The privilege privilege-level parameter specifies the privilege level for the account. You can specify one of the following:
• 0 - Super User level (full read-write access)
• 4 - Port Configuration level
• 5 - Read Only level
The default privilege level is 0 .
(FastIron security guide)
Enable is more of a "su" than a "sudo" thing...
 
May 1, 2020
37
8
8
It's not possible to use a key authentication during the enable phase because you're in a strictly text-only phase... and it must work even on a serial console that doesn't know anything about SSH keys.
You can however disable it; not the wiser move in my opinion, but in a home/lab environment... why not?
That's a good point, I'd have realized that if I gave it some more thought. I should've realized that it had to be available for everything; it's not like ssh is the only way to connect.

Makes it slightly more of a pain if you want to use a long secure password.
 

infoMatt

Active Member
Apr 16, 2019
157
60
28
That's a good point, I'd have realized that if I gave it some more thought. I should've realized that it had to be available for everything; it's not like ssh is the only way to connect.

Makes it slightly more of a pain if you want to use a long secure password.
You can still use a long-ish password for user authentication and a relatively-easier one for enable... they doesn't have to be the same.

But still, at home you don't have mandatory security policies and what not, just don't expose the admin interface.
 

EngChiSTH

Member
Jun 27, 2018
45
14
8
Chicago
That's the thing, I don't want anything complex, but no matter what I do here it doesn't seem as though I'm understanding.

All I want is 5 VLANS similar to the guide here: pfSense baseline guide with VPN, Guest and VLAN support

But this is to too much for me to do via CLI without some kind of guide. If I could get the web interface up when connected to pfSense, maybe I could do it, but I can't do it with my knowledge.
I recommend starting small and then dealing with VLANs and other advanced stuff later. i.e.
- take a piece of paper and write out what you want your network to do
- turn that piece of paper over and components that exist in your network (router/firewall, switch/switches, etc)
- then assign what services you want what device to take care of. i.e. who is assigning IP addresses on your network? pfsense? brocade switch? the device you got from your ISP? something else (i.e. NAS can run DHCP as well). who is running DNS in your network? etc

do this before you buy first piece of equipment or buy more equipment. there are many ways to do it and no single 'right answer'.

i.e. for comparison - I knew that
- I would have wired and wireless devices in my network
- my addresses are maintained by the DHCP running on W2016S. all wired devices have their IPs as static, all dynamic IPs are within specific range outside of static IP ranges.. for wired devices I 'group' them into ranges that make sense to me - NAS devices separately from streaming devices, from switch devices.
- my router is a tiny very low power consumption device from Mikrotik (RB750GR3 - Hex 3 router) , cheap and very stable for me.
- my switch is Brocade 6450 (as yours I think), it does not run any services, just does switching.
- my DNS are handled by W2016S that in turn redirects it to a Pi-hole (great software , btw) running over VM. ad-filtering.

everything just works and requires no tinkering. most of services are network services (hd homerun for tuner, obihai for phone, all data is on two NAS devices that backup to each others, steaming hardware is Nvidia Shield and FireTV , plex ,etc) . I have not run any CLI on brocade after going through great install guide from @fohdeesha and then updating firmware once. I have not felt I need Guest VLAN or anything advanced like that.

I like pfsense for capabilities and used to run it but think it is great overall for what I actually want things to do.
 

Jiaxinxi

Member
Aug 23, 2017
69
29
18
50
First of all thanks for @fohdeesha excellent document.

I am able to flush my ICX7250-48 to 8080E firmware and enable all 8x10G port, L3 features.

The fan come with the switch is pretty quiet when system boots up, but it is not quiet enough to be in living room during late night.

So the fan needs to be changed. ICX7250-48 has two fans. I switched to HA40201V3-A000-G99. Once the system bootsup, it is dead silent.

However, there is not enough air flow to cool the main chip. so after few minutes, the fan will run in full speed which is pretty noisy.

I searched online, found GDSTIME Quiet 80mm Fan, 12V 3PIN 80mm x 80mm x 10mm Brushless DC Fan for CPU Coolers , put on top of the chip heat sink and it fits perfectly.

1590471116154.png

The temperature is better. Fan will not run in full speed any more under normal load.
1590509223352.png
 
  • Like
Reactions: epicurean

John Francini

New Member
Jan 4, 2020
5
2
3
Question on a Brocade ICX-6610. I bought it several months ago, and did some initial configuration on the switch, and then due to Real Life getting in the way, powered off the switch and left it alone for several months.

Last night I came back to the switch, powered it on, and it won't boot. I get a lot of I2C errors, and eventually it reports that one of the fan modules (right-hand side) is failed and it won't boot with the failed module. So just for a lark I swapped the fan modules from one side to the other, figuring that if the module was bad the problem would move.

It didn't.

The right-hand side fan module still reports the error -- and in fact, it is not spinning. Swap it to the other side and it spins just fine.

Do I need to open the switch up and look for things like cold solder joints and the like? Or could it be a power supply problem?

I don't have the serial port log handy; I can go generate one and post it here if it would help.

Thanks,

John

Log below:

Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files........
error in reading i2c device 1
eeprom_read: reading i2c device 1 failed
lost arbitration, dev=52
i2c controller reset, dev=52
error in writing i2c device 1, error code: 4
eeprom_write: writing i2c device 1 failed
error in reading i2c device 1
ONBOARD_EEPROM read failure 1.
Backplane MAC address is not configured.
Use default 0004.80a0.4000
BOOT INFO: load image from primary copy...

error in reading i2c device 1, error code 5
eeprom_read: reading i2c device 1 failed
lost arbitration, dev=52
i2c controller reset, dev=52
error in writing i2c device 1, error code: 4
eeprom_write: writing i2c device 1 failed
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
..CPSS DxCh Version: cpss3.4p1 release
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 5, ps 1
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 2
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 4, ps 2
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 3
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=2f
i2c controller reset, dev=2f
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=2f
i2c controller reset, dev=2f
lost arbitration, dev=2f
i2c controller reset, dev=2f
Fatal Error: No FANTRAY fan is present or fan has failed
System is shutting down!!!
Rebooting(0)...
*
$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
.
 
Last edited:

infoMatt

Active Member
Apr 16, 2019
157
60
28
Question on a Brocade ICX-6610. I bought it several months ago, and did some initial configuration on the switch, and then due to Real Life getting in the way, powered off the switch and left it alone for several months.

Last night I came back to the switch, powered it on, and it won't boot. I get a lot of I2C errors, and eventually it reports that one of the fan modules (right-hand side) is failed and it won't boot with the failed module. So just for a lark I swapped the fan modules from one side to the other, figuring that if the module was bad the problem would move.

It didn't.

The right-hand side fan module still reports the error -- and in fact, it is not spinning. Swap it to the other side and it spins just fine.
Could it be due to some oxidation on the module control pins? Try booting the switch with only one fan module, and see if it complains.
 

klui

Active Member
Feb 3, 2019
160
67
28
Last night I came back to the switch, powered it on, and it won't boot. I get a lot of I2C errors, and eventually it reports that one of the fan modules (right-hand side) is failed and it won't boot with the failed module. So just for a lark I swapped the fan modules from one side to the other, figuring that if the module was bad the problem would move.
The log showing an issue with PSU 3 is not right. You're also getting something from "chow_led_ch_mux_enable" means it's a lower level issue from the ASIC. Search for "chow" in this thread and check out @fohdeesha's analysis.

You should definitely follow @infoMatt's advice and clean out your PSU/fan connectors. Install one set of fan/PSU on both bays to see what parts are bad.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
Question on a Brocade ICX-6610. I bought it several months ago, and did some initial configuration on the switch, and then due to Real Life getting in the way, powered off the switch and left it alone for several months.

Last night I came back to the switch, powered it on, and it won't boot. I get a lot of I2C errors, and eventually it reports that one of the fan modules (right-hand side) is failed and it won't boot with the failed module. So just for a lark I swapped the fan modules from one side to the other, figuring that if the module was bad the problem would move.

It didn't.

The right-hand side fan module still reports the error -- and in fact, it is not spinning. Swap it to the other side and it spins just fine.

Do I need to open the switch up and look for things like cold solder joints and the like? Or could it be a power supply problem?

I don't have the serial port log handy; I can go generate one and post it here if it would help.

Thanks,

John

Log below:

Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files........
error in reading i2c device 1
eeprom_read: reading i2c device 1 failed
lost arbitration, dev=52
i2c controller reset, dev=52
error in writing i2c device 1, error code: 4
eeprom_write: writing i2c device 1 failed
error in reading i2c device 1
ONBOARD_EEPROM read failure 1.
Backplane MAC address is not configured.
Use default 0004.80a0.4000
BOOT INFO: load image from primary copy...

error in reading i2c device 1, error code 5
eeprom_read: reading i2c device 1 failed
lost arbitration, dev=52
i2c controller reset, dev=52
error in writing i2c device 1, error code: 4
eeprom_write: writing i2c device 1 failed
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
..CPSS DxCh Version: cpss3.4p1 release
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 5, ps 1
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 2
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 4, ps 2
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 3
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=2f
i2c controller reset, dev=2f
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=2f
i2c controller reset, dev=2f
lost arbitration, dev=2f
i2c controller reset, dev=2f
Fatal Error: No FANTRAY fan is present or fan has failed
System is shutting down!!!
Rebooting(0)...
*
$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
.

something is screwing up the i2c bus on the switch and it can no longer read anything over it (the PSUs or the fan controller for example). Is this a POE model? it could be one of the power supplies, if you have two, try just one at a time and see if the error goes away with one of them.
 

John Francini

New Member
Jan 4, 2020
5
2
3
Okay. I've tried swapping the power supplies from one side to the other, and powering only one at a time, and still the same errors. One thing I note is that the fans on the right side actually do spin, but not at full speed, unlike their screaming cousins on the left side. Swapping the fan modules doesn't help - now the screamer is indolent and the indolent one is screaming.

Sounds like something's definitely wrong in that switch. Time for a replacement - I've already ordered a 7250, since I don't need all that many gigabit ports, but I definitely want the 8x10gig ports.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
I mean actually remove them from the chassis and try each psu inserted only one at a time. When they're in the slot and not powered they're still connected to the i2c bus
 

John Francini

New Member
Jan 4, 2020
5
2
3
When I said I powered them one at a time, the non-powered ones were pulled out 2-3 inches so they were not engaged in their power sockets. Sorry I wasn't clear.
 
  • Like
Reactions: fohdeesha

m4r1k

Member
Nov 4, 2016
52
6
8
31
Hey guys,

Brocade is giving me a hard time with certain PXE scenario where I keep hitting the following error

Code:
dynamic lag 10 peer info (priority=65535,id=ecf4.bbdd.9628,key=1) remove (LagExpiry)
Basically the LAG goes up as per force-up config, it stays up for maybe 10/20 seconds and then all the ports in the lag go in blocked state.

Here the lag config

Same environment using RHEL8 and it works like a charm. Indeed the issue could be RHEL7 related but maybe somebody has a clue.
Thanks!
 

infoMatt

Active Member
Apr 16, 2019
157
60
28
Hey guys,

Brocade is giving me a hard time with certain PXE scenario where I keep hitting the following error

Code:
dynamic lag 10 peer info (priority=65535,id=ecf4.bbdd.9628,key=1) remove (LagExpiry)
Basically the LAG goes up as per force-up config, it stays up for maybe 10/20 seconds and then all the ports in the lag go in blocked state.

Here the lag config

Same environment using RHEL8 and it works like a charm. Indeed the issue could be RHEL7 related but maybe somebody has a clue.
Thanks!
Make sure that you've configured the correct options in the bonding interfaces... try something like "mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer2+3".
If you don't specify the 802.3d mode it won't send LACP control frames down the wire and thus the switch drops the LAG 'cause it doesn't detect a correct configured peer and it won't risk dropping traffic or worse creating a network loop.