Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[BenjaminCalibration]

It's not possible to use a key authentication during the enable phase because you're in a strictly text-only phase... and it must work even on a serial console that doesn't know anything about SSH keys.
You can however disable it; not the wiser move in my opinion, but in a home/lab environment... why not?

That's a good point, I'd have realized that if I gave it some more thought. I should've realized that it had to be available for everything; it's not like ssh is the only way to connect.

Makes it slightly more of a pain if you want to use a long secure password.

 

[infoMatt]

That's a good point, I'd have realized that if I gave it some more thought. I should've realized that it had to be available for everything; it's not like ssh is the only way to connect.

Makes it slightly more of a pain if you want to use a long secure password.

You can still use a long-ish password for user authentication and a relatively-easier one for enable... they doesn't have to be the same.

But still, at home you don't have mandatory security policies and what not, just don't expose the admin interface.

 

[fohdeesha]

still looking to buy PSU blank covers if anyone has one or two (ICX6610)

 

[EngChiSTH]

That's the thing, I don't want anything complex, but no matter what I do here it doesn't seem as though I'm understanding.

All I want is 5 VLANS similar to the guide here: pfSense baseline guide with VPN, Guest and VLAN support

But this is to too much for me to do via CLI without some kind of guide. If I could get the web interface up when connected to pfSense, maybe I could do it, but I can't do it with my knowledge.

I recommend starting small and then dealing with VLANs and other advanced stuff later. i.e.
- take a piece of paper and write out what you want your network to do
- turn that piece of paper over and components that exist in your network (router/firewall, switch/switches, etc)
- then assign what services you want what device to take care of. i.e. who is assigning IP addresses on your network? pfsense? brocade switch? the device you got from your ISP? something else (i.e. NAS can run DHCP as well). who is running DNS in your network? etc

do this before you buy first piece of equipment or buy more equipment. there are many ways to do it and no single 'right answer'.

i.e. for comparison - I knew that
- I would have wired and wireless devices in my network
- my addresses are maintained by the DHCP running on W2016S. all wired devices have their IPs as static, all dynamic IPs are within specific range outside of static IP ranges.. for wired devices I 'group' them into ranges that make sense to me - NAS devices separately from streaming devices, from switch devices.
- my router is a tiny very low power consumption device from Mikrotik (RB750GR3 - Hex 3 router) , cheap and very stable for me.
- my switch is Brocade 6450 (as yours I think), it does not run any services, just does switching.
- my DNS are handled by W2016S that in turn redirects it to a Pi-hole (great software , btw) running over VM. ad-filtering.

everything just works and requires no tinkering. most of services are network services (hd homerun for tuner, obihai for phone, all data is on two NAS devices that backup to each others, steaming hardware is Nvidia Shield and FireTV , plex ,etc) . I have not run any CLI on brocade after going through great install guide from @fohdeesha and then updating firmware once. I have not felt I need Guest VLAN or anything advanced like that.

I like pfsense for capabilities and used to run it but think it is great overall for what I actually want things to do.

 

[Jiaxinxi]

First of all thanks for @fohdeesha excellent document.

I am able to flush my ICX7250-48 to 8080E firmware and enable all 8x10G port, L3 features.

The fan come with the switch is pretty quiet when system boots up, but it is not quiet enough to be in living room during late night.

So the fan needs to be changed. ICX7250-48 has two fans. I switched to HA40201V3-A000-G99. Once the system bootsup, it is dead silent.

However, there is not enough air flow to cool the main chip. so after few minutes, the fan will run in full speed which is pretty noisy.

I searched online, found GDSTIME Quiet 80mm Fan, 12V 3PIN 80mm x 80mm x 10mm Brushless DC Fan for CPU Coolers , put on top of the chip heat sink and it fits perfectly.



The temperature is better. Fan will not run in full speed any more under normal load.

 

[John Francini]

Question on a Brocade ICX-6610. I bought it several months ago, and did some initial configuration on the switch, and then due to Real Life getting in the way, powered off the switch and left it alone for several months.

Last night I came back to the switch, powered it on, and it won't boot. I get a lot of I2C errors, and eventually it reports that one of the fan modules (right-hand side) is failed and it won't boot with the failed module. So just for a lark I swapped the fan modules from one side to the other, figuring that if the module was bad the problem would move.

It didn't.

The right-hand side fan module still reports the error -- and in fact, it is not spinning. Swap it to the other side and it spins just fine.

Do I need to open the switch up and look for things like cold solder joints and the like? Or could it be a power supply problem?

I don't have the serial port log handy; I can go generate one and post it here if it would help.

Thanks,

John

Log below:

Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files........
error in reading i2c device 1
eeprom_read: reading i2c device 1 failed
lost arbitration, dev=52
i2c controller reset, dev=52
error in writing i2c device 1, error code: 4
eeprom_write: writing i2c device 1 failed
error in reading i2c device 1
ONBOARD_EEPROM read failure 1.
Backplane MAC address is not configured.
Use default 0004.80a0.4000
BOOT INFO: load image from primary copy...

error in reading i2c device 1, error code 5
eeprom_read: reading i2c device 1 failed
lost arbitration, dev=52
i2c controller reset, dev=52
error in writing i2c device 1, error code: 4
eeprom_write: writing i2c device 1 failed
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
..CPSS DxCh Version: cpss3.4p1 release
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 5, ps 1
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 2
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 4, ps 2
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 3
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=2f
i2c controller reset, dev=2f
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=2f
i2c controller reset, dev=2f
lost arbitration, dev=2f
i2c controller reset, dev=2f
Fatal Error: No FANTRAY fan is present or fan has failed
System is shutting down!!!
Rebooting(0)...
*
$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
.

 

[infoMatt]

Question on a Brocade ICX-6610. I bought it several months ago, and did some initial configuration on the switch, and then due to Real Life getting in the way, powered off the switch and left it alone for several months.

Last night I came back to the switch, powered it on, and it won't boot. I get a lot of I2C errors, and eventually it reports that one of the fan modules (right-hand side) is failed and it won't boot with the failed module. So just for a lark I swapped the fan modules from one side to the other, figuring that if the module was bad the problem would move.

It didn't.

The right-hand side fan module still reports the error -- and in fact, it is not spinning. Swap it to the other side and it spins just fine.

Could it be due to some oxidation on the module control pins? Try booting the switch with only one fan module, and see if it complains.

 

[klui]

Last night I came back to the switch, powered it on, and it won't boot. I get a lot of I2C errors, and eventually it reports that one of the fan modules (right-hand side) is failed and it won't boot with the failed module. So just for a lark I swapped the fan modules from one side to the other, figuring that if the module was bad the problem would move.

The log showing an issue with PSU 3 is not right. You're also getting something from "chow_led_ch_mux_enable" means it's a lower level issue from the ASIC. Search for "chow" in this thread and check out @fohdeesha's analysis.

You should definitely follow @infoMatt's advice and clean out your PSU/fan connectors. Install one set of fan/PSU on both bays to see what parts are bad.

 

[fohdeesha]

Question on a Brocade ICX-6610. I bought it several months ago, and did some initial configuration on the switch, and then due to Real Life getting in the way, powered off the switch and left it alone for several months.

Last night I came back to the switch, powered it on, and it won't boot. I get a lot of I2C errors, and eventually it reports that one of the fan modules (right-hand side) is failed and it won't boot with the failed module. So just for a lark I swapped the fan modules from one side to the other, figuring that if the module was bad the problem would move.

It didn't.

The right-hand side fan module still reports the error -- and in fact, it is not spinning. Swap it to the other side and it spins just fine.

Do I need to open the switch up and look for things like cold solder joints and the like? Or could it be a power supply problem?

I don't have the serial port log handy; I can go generate one and post it here if it would help.

Thanks,

John

Log below:

Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files........
error in reading i2c device 1
eeprom_read: reading i2c device 1 failed
lost arbitration, dev=52
i2c controller reset, dev=52
error in writing i2c device 1, error code: 4
eeprom_write: writing i2c device 1 failed
error in reading i2c device 1
ONBOARD_EEPROM read failure 1.
Backplane MAC address is not configured.
Use default 0004.80a0.4000
BOOT INFO: load image from primary copy...

error in reading i2c device 1, error code 5
eeprom_read: reading i2c device 1 failed
lost arbitration, dev=52
i2c controller reset, dev=52
error in writing i2c device 1, error code: 4
eeprom_write: writing i2c device 1 failed
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
..CPSS DxCh Version: cpss3.4p1 release
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 5, ps 1
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 2
lost arbitration, dev=70
i2c controller reset, dev=70
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 4, ps 2
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 3
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=2f
i2c controller reset, dev=2f
lost arbitration, dev=70
i2c controller reset, dev=70
lost arbitration, dev=2f
i2c controller reset, dev=2f
lost arbitration, dev=2f
i2c controller reset, dev=2f
Fatal Error: No FANTRAY fan is present or fan has failed
System is shutting down!!!
Rebooting(0)...
*
$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
.

something is screwing up the i2c bus on the switch and it can no longer read anything over it (the PSUs or the fan controller for example). Is this a POE model? it could be one of the power supplies, if you have two, try just one at a time and see if the error goes away with one of them.

 

[John Francini]

Okay. I've tried swapping the power supplies from one side to the other, and powering only one at a time, and still the same errors. One thing I note is that the fans on the right side actually do spin, but not at full speed, unlike their screaming cousins on the left side. Swapping the fan modules doesn't help - now the screamer is indolent and the indolent one is screaming.

Sounds like something's definitely wrong in that switch. Time for a replacement - I've already ordered a 7250, since I don't need all that many gigabit ports, but I definitely want the 8x10gig ports.

 

[fohdeesha]

I mean actually remove them from the chassis and try each psu inserted only one at a time. When they're in the slot and not powered they're still connected to the i2c bus

 

[John Francini]

When I said I powered them one at a time, the non-powered ones were pulled out 2-3 inches so they were not engaged in their power sockets. Sorry I wasn't clear.

 

[fohdeesha]

ok then yeah it's ****ed

 

[fohdeesha]

odd question, did you happen to dust the thing out previously by blowing canned air into the fan trays?

 

[m4r1k]

Hey guys,

Brocade is giving me a hard time with certain PXE scenario where I keep hitting the following error

dynamic lag 10 peer info (priority=65535,id=ecf4.bbdd.9628,key=1) remove (LagExpiry)

Basically the LAG goes up as per force-up config, it stays up for maybe 10/20 seconds and then all the ports in the lag go in blocked state.

Here the lag config

m4r1k/nfvi_lab

NFVi Performance Lab using Red Hat OpenStack. Contribute to m4r1k/nfvi_lab development by creating an account on GitHub.

github.com

Same environment using RHEL8 and it works like a charm. Indeed the issue could be RHEL7 related but maybe somebody has a clue.
Thanks!

 

[infoMatt]

Hey guys,

Brocade is giving me a hard time with certain PXE scenario where I keep hitting the following error

dynamic lag 10 peer info (priority=65535,id=ecf4.bbdd.9628,key=1) remove (LagExpiry)

Basically the LAG goes up as per force-up config, it stays up for maybe 10/20 seconds and then all the ports in the lag go in blocked state.

Here the lag config

m4r1k/nfvi_lab

NFVi Performance Lab using Red Hat OpenStack. Contribute to m4r1k/nfvi_lab development by creating an account on GitHub.

github.com

Same environment using RHEL8 and it works like a charm. Indeed the issue could be RHEL7 related but maybe somebody has a clue.
Thanks!

Make sure that you've configured the correct options in the bonding interfaces... try something like "mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer2+3".
If you don't specify the 802.3d mode it won't send LACP control frames down the wire and thus the switch drops the LAG 'cause it doesn't detect a correct configured peer and it won't risk dropping traffic or worse creating a network loop.

 

[m4r1k]

Make sure that you've configured the correct options in the bonding interfaces... try something like "mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer2+3".
If you don't specify the 802.3d mode it won't send LACP control frames down the wire and thus the switch drops the LAG 'cause it doesn't detect a correct configured peer and it won't risk dropping traffic or worse creating a network loop.

Thanks, indeed LACP options are correct [1]. The problem doesn't happen when Linux is happily up and running but during the PXE phase (hence also the force-up)

I cannot find anywhere meaning for LagExpiry (the issue only persist with RHEL7 while RHEL8 is fine but of course it might be a timing problem and having a faster PXE like the one in RHEL8 won't trigger it)


[1] m4r1k/nfvi_lab

 

[infoMatt]

Thanks, indeed LACP options are correct [1]. The problem doesn't happen when Linux is happily up and running but during the PXE phase (hence also the force-up)

I don't think it's a safe choice to boot from a LACP... or at least, you could boot with just one interface of the bond up and then hot plug the other.

"By feeling", I've assumed that a LagExpired means that it couldn't receive the LACP renew control packet before the refresh timer expires... It could be that it cannot apply the LACP configuration correctly to a interface that is already UP and from where it booted...

EDIT: @m4r1k take a look at another user with a similar problem using FreeNAS: https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-223952

 

[m4r1k]

I don't think it's a safe choice to boot from a LACP... or at least, you could boot with just one interface of the bond up and then hot plug the other.

PXE boot over LACP might not be common with Brocade but is done by plenty of people, you just the correct config, like NXOS LACP set in passive mode or the force-up of Brocade and Juniper ;-)
The config is also described in the ICX manuals, so nothing esoteric here

"By feeling", I've assumed that a LagExpired means that it couldn't receive the LACP renew control packet before the refresh timer expires... It could be that it cannot apply the LACP configuration correctly to a interface that is already UP and from where it booted...

Good point! In the PXE process, nothing that I know tried to create LACP, everything happens after this phase and yet it goes down.

Spoiler: Here the full logs just generated

May 27 16:17:29:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is down.
May 27 16:17:28:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is down.
May 27 16:17:28:I:System: Logical link on dynamic lag interface ethernet 1/1/14 is down.
May 27 16:17:25:I:System: dynamic lag 10 peer info (priority=65535,id=ecf4.bbdd.9628,key=1) remove (LagExpiry) <= the issue
May 27 16:17:25:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is down.
May 27 16:17:25:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is down.
May 27 16:17:25:I:System: Logical link on dynamic lag interface ethernet 1/1/14 is down.
May 27 16:17:25:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is down.
May 27 16:17:22:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is down.
May 27 16:17:22:I:Trunk: Group (1/1/13) removed by 802.3ad link-aggregation module.
May 27 16:17:22:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is up.<= starting of the PXE process
May 27 16:17:19:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is down.
May 27 16:17:19:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is down.
May 27 16:17:19:I:System: Logical link on dynamic lag interface ethernet 1/1/14 is down.
May 27 16:15:18:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is up.
May 27 16:15:18:I:System: Interface ethernet 1/1/13, state up
May 27 16:15:16:I:System: Interface ethernet 1/1/13, state down
May 27 16:15:13:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is up.
May 27 16:15:13:I:System: Interface ethernet 1/1/13, state up <= during the POST
May 27 16:15:10:I:System: Interface ethernet 1/1/16, state down
May 27 16:15:10:I:System: Interface ethernet 1/1/15, state down
May 27 16:15:09:I:System: Interface ethernet 1/1/14, state down

Spoiler: Here the LAG details at the beginning of the PXE process

SSH@diablo(config-lag-R630)#show lag R630
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (117 available)
LACP System Priority / ID: 1 / cc4e.24f8.98d0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3

=== LAG "R630" ID 10 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/13 to 1/1/16
Port Count: 4
Primary Port: 1/1/13
Trunk Type: hash-based
LACP Key: 20010
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/13 Up Forward Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/14 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/15 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/16 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/1/13 1 1 20010 Yes L Agg Syn Col Dis No No Ope
1/1/14 1 1 20010 Yes L Agg Syn No No Def Exp Ina
1/1/15 1 1 20010 Yes L Agg Syn No No Def Exp Ina
1/1/16 1 1 20010 Yes L Agg Syn No No Def Exp Ina


Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/1/13 65535-ecf4.bbdd.9628 1 4 20
1/1/14 1-0000.0000.0000 13 0 91
1/1/15 1-0000.0000.0000 14 0 91
1/1/16 1-0000.0000.0000 15 0 92

Spoiler: Here the LAG details at the beginning of the PXE process

SSH@diablo(config-lag-R630)#show lag R630
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (117 available)
LACP System Priority / ID: 1 / cc4e.24f8.98d0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3

=== LAG "R630" ID 10 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/13 to 1/1/16
Port Count: 4
Primary Port: 1/1/13
Trunk Type: hash-based
LACP Key: 20010
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/13 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/14 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/15 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/16 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/1/13 1 1 20010 Yes S Agg Syn No No Def Exp Frc
1/1/14 1 1 20010 Yes S Agg Syn Col Dis Def No Ina
1/1/15 1 1 20010 Yes S Agg Syn Col Dis Def No Ina
1/1/16 1 1 20010 Yes S Agg Syn Col Dis Def No Ina


Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/1/13 1-0000.0000.0000 12 4 437
1/1/14 1-0000.0000.0000 13 0 576
1/1/15 1-0000.0000.0000 14 0 576
1/1/16 1-0000.0000.0000 15 0 578

 

[m4r1k]

Well, weird enough, specifying 'lacp-timeout short' the lag actually stays up.
I'll now run the entire deployment process and report back. Maybe somebody else ran into the same issue