Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

that is incredibly odd - I don't know how reflashing firmware would have fixed it (unless it was a different version). The OS image is checksummed entirely at boot so if there was anything corrupt with it, it would refuse to boot (or fall back to secondary OS slot)
when you say remove stack units, do you mean the lines in the config? Remove those, reload switch, and let me know if the behavior returns - that's the only difference between the old setup that was failing and the new one that isn't, as far as I can tell

 

[N1h1l1ty]

I've removed the stack units in the config, booted it (with the old DAC cables that I initially thought were the root cause) and it has not thrown any error:



Very odd indeed. I can't think of anything that I had done that would have thrown those errors in the first place... Now to test out if the throughput issue is still there!

 

[N1h1l1ty]

@fohdeesha - OK I've managed to recreate the error again - Initially I thought it might be related to aaa authentication settings but it seems that its VLAN related - when I have ports 1/2/7 and 1/2/2 assigned to any vlans, be they tagged or untagged, the error crops up again

 

[fohdeesha]

 

[N1h1l1ty]

Anything else I should try at this point to try and narrow it down?

With no VLAN tagging throughput looks normal and im no longer seeing the crazy jitter / packet out of order messages on iperf.

EDIT: Yep final confirmation here - looks like if the ports are used as access ports, everything works fine. However, as soon as I introduced tagging, the interfaces shut down and upon reloading the switch it is once again throwing the error.

I've tried this now on a range of firmware, from 8030sa to 8030r and 8020

 

[fohdeesha]

that honestly sounds like bad hardware/bad 40g stack card, never seen anything like that before. Can you post your config?

And you're sure it's not an issue with the vlan handling on the end devices, right?

 

[N1h1l1ty]

that honestly sounds like bad hardware/bad 40g stack card, never seen anything like that before. Can you post your config?

And you're sure it's not an issue with the vlan handling on the end devices, right?

Definitely not an issue of vlan handling on the end devices as I've checked them before with the exact same configuration on my Cisco 3750E with no throughput issues - furthermore I just rechecked the config with ports 1/2/2 through 1/2/5 and 1/2/7 through 1/2/10 untagged on a vlan and had no issues achieving full 10g throughput.

Config:

Current configuration:
!
ver 08.0.20T7f3
!
stack unit 1
module 1 icx6610-24p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 10 name Management by port
tagged ethe 1/1/24
untagged ethe 1/1/1 to 1/1/6
spanning-tree
!
vlan 20 name Hosts by port
tagged ethe 1/1/24
untagged ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10
spanning-tree
!
!
!
!
!
ip dhcp-client disable
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
end

VLANs:

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree On
Untagged Ports: (U1/M1) 7 8 9 10 11 12 13 14 15 16 17 18
Untagged Ports: (U1/M1) 19 20 21 22 23
Untagged Ports: (U1/M2) 1 6
Untagged Ports: (U1/M3) 1 2 3 4 5 6 7 8
Tagged Ports: None
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled
PORT-VLAN 10, Name Management, Priority level0, Spanning tree On
Untagged Ports: (U1/M1) 1 2 3 4 5 6
Tagged Ports: (U1/M1) 24
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled
PORT-VLAN 20, Name Hosts, Priority level0, Spanning tree On
Untagged Ports: (U1/M2) 2 3 4 5 7 8 9 10
Tagged Ports: (U1/M1) 24
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

 

[N1h1l1ty]

Let me configure tagged again and post that

 

[N1h1l1ty]

Yep, as expected, we're right back to this:

Parsing Config Data ...
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/3 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/3
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/4 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/4
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/5 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/5

(only have 1 QSFP breakout cable attached at the moment, but I'm betting it would do the same for the second cable)

Worth noting I only have port 1/2/2 and 1/2/3 plugged in.

ICX6610-24P Router>show run
Current configuration:
!
ver 08.0.20T7f3
!
stack unit 1
module 1 icx6610-24p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree
!
vlan 10 name Management by port
tagged ethe 1/1/24
untagged ethe 1/1/1 to 1/1/6
spanning-tree
!
vlan 20 name Hosts by port
tagged ethe 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10
spanning-tree
!
!
!
!
!
ip dhcp-client disable
!
!
!
!
!
!
!
!
!
!
interface ve 1
ip address 192.168.0.55 255.255.255.0
!
!
!
!
!
!
!
!
!
end

Is current running config.

 

[fohdeesha]

that is really bizarre. just out of curiosity, can you remove any refs to stp and reload the switch, and check for same behavior?

 

[N1h1l1ty]

Had that same thought actually, will try to do that now - that being said, it gets even *more* bizarre. I swear I am not making this up but in the meantime I've continued to troubleshoot, and I'm now seeing correct throughput across both servers with tagged vlans. Now, it could have been configuration error on my end because I completely rebuilt the client configurations in Windows Server 16 from scratch again, but what I *did* notice is that the behavior where 1/2/2 through 1/2/5 and 1/2/7 through 1/2/7 will shut down if I make any tagged vlan changes to them, and also shut down and will not come back up after being added to dynamic LAGs persists, despite me no longer seeing jitter / loss and limited bandwidth across a single link - I paired 1/2/2 - 1/2/3 and 1/2/4 - 1/2/5 and when I deployed each LAG, both ports would shut down requiring me to reload the switch.

 

[fohdeesha]

I think barring an act of god, you have a defective switch. the "M:9 L:0 - chow_qsfp_read, qsfp 2, error" are low level ASIC messages (Chow is the codename for this particular model of Marvell Prestera silicon). You should never see error messages from the ASIC itself unless something has gone horribly wrong

Aside from everyone else in the thread, I know I've personally used the rear breakout ports tagged in a ton of vlans at once while also in LAGs (even LAG's involving breakout ports from 2 different stack members). Did so to serve a LAN party of 150+ people without issue: Fohdeesha/reboot-lan

 

[N1h1l1ty]

Yeah unfortunately that may just be the case. I'm going to test the 10gbe SFP+ modules tomorrow or the day after since I don't have any additional optics handy to get them under proper load. That being said, if everything else works I'll probably hang on to it and get another one down the line - And for what its worth, I'm seeing the breakout ports function more or less as expected right now - 10 gigabit throughput with correct LAG configuration end to end. I'll try and do some more testing in the morning as I've only got them tagged with 1 VLAN right now - I had them tagged with 3 before and that's where I was seeing issues...

EDIT: Meant front SFP+ modules

 

[mattaw]

So, having purchased 3x ICX6610-48P as-is, expecting only one to work, I find that they all seem to work fine. I have used @fohdeesha 's excellent guide and updated the bootloader, firmware and preliminarily tested a few gigabit ports on the front.

I wonder if there is a built in self-test routine I can run somewhere? I am assuming there are a bunch of checks it runs on boot? I would love to give the switches a full test. I have ordered an active poe splitter as that is the only thing I think would enable a proper test of the POE functions.

Also one of the systems identifies itself as STACKID 2 even after factory set-defaults. Is this normal?

 

[svtkobra7]

Received my ICX6450-24 yesterday and thanks to @fohdeesha 's excellent documentation was able to get it up and running in short order.

In an attempt to contribute to the community, I figured it was relevant to call out a certain ebay listing for anyone looking to purchase (note I'm in no way associated with the seller): (ICX6450-24) BROCADE ICX 6450-24 SWITCH 24 PORTS MANAGED DESKTOP RACK/WALL-MOUNT 783555168134 | eBay

Before you gawk at the $170 price, note that I'm the "1 sold" (2 available) and the seller quickly accepted my "best offer" via ebay message for $150 (and then changed listing format to accept BOs). Personally I thought $150 was a tad high until I opened up my new toy, then I thought it was a very good one:

1. Listing = "gently used," but it is in absolutely mint condition (no visible sign of it ever being touched);
   
2. Original box (opened however);
3. Accessory kit with all sealed bags (rack ears, screws for rack ears, console cable, rack screws, and rack nuts);
4. The power cable still has the original twist tie on it (o/c it could have been used with another cable).

I figure just the console cable may be worth $10 to someone (note it is serial not USB though) and good luck finding rack ears separately a decent price.

So, as I said just a small attempt to give back for anyone who is currently looking for a ICX6450-24 and wants to now exactly what they are getting. Considering my offer @ $150 was accepted without hesitation, I'd probably go lower initially.

 

[fohdeesha]

So, having purchased 3x ICX6610-48P as-is, expecting only one to work, I find that they all seem to work fine. I have used @fohdeesha 's excellent guide and updated the bootloader, firmware and preliminarily tested a few gigabit ports on the front.

I wonder if there is a built in self-test routine I can run somewhere? I am assuming there are a bunch of checks it runs on boot? I would love to give the switches a full test. I have ordered an active poe splitter as that is the only thing I think would enable a proper test of the POE functions.

Also one of the systems identifies itself as STACKID 2 even after factory set-defaults. Is this normal?

on boot it does pretty thorough checking of RAM, the ASIC(s), the i2c bus, etc. There's a dedicated test routine hidden in the bootloader if I remember right but it involves a snake test (daisychaining every single port to the next one)

If you ran factory set-default in the old bootloader it shipped with, it's possible it was old enough it didn't remove stacking config as part of it's reset routine. You can try running it again now that it's on the latest BL

or from the OS enable level you can try running "stack unconfigure clean" (if its not stack master, you need to do stack unconfigure me)

if that STILL doesn't work, hard mode:
over serial (will not work over telnet/ssh)
#at any OS level:
#press ctrl+y, let go, then press m, then hit enter
#at the new console, show files with:
dir
#delete all the config files
del stacking.boot
del startup-config
del startup-config.backup
del startup-config.old
#reboot
reset

 

[svtkobra7]

And here I've been thinking all these years a "snake test" was the ultimate test of mobile phone mastery ...

 

[fohdeesha]

7150 and 7250 guides/firmware up: FCX / ICX6610 - Fohdeesha Docs
(on left hand side)
Also gave it a makeover (changed the color)

as usual if something is confusing or doesn't work let me know

PS: when I made this thread I did not think we'd make it to 5 pages, and now we're at page 50 three or four months later. Crazy! Hope everyone is enjoying their gear - I know I felt like a kid in a candyshop when I found these models and how well they work for the price, glad to share that with others

 

[kapone]

So..uh..a funny thing happened.

A 6610-24P popped up on ebay, it had minor bending of two of the SFP ports, and had NO power supplies or fans, just the chassis. I looked at it and said, what a waste...but then...the 24P does have a lower power draw than the 48 port version, and I don't care about the ethernet ports as much, I use may be 10 of them (the IPMI network is on it's own switch). I care about the 10g/40g ports and that's the same between the two versions.

So...I put in a offer of $40 (including shipping) just for the heck of it. My thinking was, I have enough power supplies and fan trays for these switches...

And...the seller accepted.

That's with shipping...I can't ship one of these for less than $20 anywhere, and I have a significant discount from Fedex.

All I can say is...

 

[fohdeesha]

this is me hiding my jealousy