Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[DASHIP]

Be careful making sure you check the model number before buying. The model numbering scheme is confusing. I just purchased a 6610-48-PE thinking it was a 6610-48P-E. The latter has PoE ports, while the former does not. If you are unsure, be sure to check the datasheets listed on the first page of this thread. Here is an excerpt for the 6610-48 as an example. Notice how close some of the model numbers are. With the noted inaccuracy of eBay postings, it is easy to get the wrong switch. Also, the models with PoE have a "P" at the end of the model name on the front of the switch, in white lettering: "ICX 6610-48P". If the "P" is not present on the front, it is not a PoE model.

 

[ArmedAviator]

tl;dr is weird without ANY change, this morning re-applying my config (same as the one on GitHub) the issue is essentially gone (well, I of course upgraded to the latest firmware but two days ago the situation was unable)

I suspect the issue is and never was with the switch but either a bad connection/cable somewhere or a routing table on an edge device doing funky things. Be sure to check for packet errors on your end device(s).

I also created a much simpler setup: two 10Gbps interfaces of two different R630 in the same vlan, ping between the two and no drop nor delay whatsoever.

This is what really matters.

Should I take that the switch is about to die?

Certainly not. Traffic sent to the switch management interface (i.e. ICMP ping) is removed from the normal path through the ASICs and sent to the management CPU. It is also fairly low priority, so if the management CPU is doing something else, the ICMP pings/SNMP returns will end up with varied latency. Meanwhile, traffic passing through the switch normally from device to device will experience none of the latency that the management CPU is returning in ping results.

 

[m4r1k]

I suspect the issue is and never was with the switch but either a bad connection/cable somewhere or a routing table on an edge device doing funky things. Be sure to check for packet errors on your end device(s).


This is what really matters.


Certainly not. Traffic sent to the switch management interface (i.e. ICMP ping) is removed from the normal path through the ASICs and sent to the management CPU. It is also fairly low priority, so if the management CPU is doing something else, the ICMP pings/SNMP returns will end up with varied latency. Meanwhile, traffic passing through the switch normally from device to device will experience none of the latency that the management CPU is returning in ping results.

I think we're speaking too early without an important variable: time. The problem is back :-(

To make sure there is nothing wrong, I re-initialized two of my baremetal nodes (an R630 and an R730) with a super simple setup (no lag, no lacp, no lldp, no loop-detection etc), a single vlan, and not config whatsoever but the result is packet drop and delay.

Tomorrow I'll run a DPDK app to check the device's stats and put some real load on the system and see the true stability beyond ping.

Last week when I opened the ICX I immediately noticed that the main CPU was wayyyy hotter than the reported 50C. Would that be something to look into?

 

[dennisp]

I'm 94 pages in and already ordered a pair of 6610 to replace the Cisco 3750 I use at home. Also I get to learn some new tech and hopefully save some power in the process. 100+ pages to go but I skipped ahead to say thanks to @fohdeesha for sharing all of this info.

 

[fohdeesha]

I think we're speaking too early without an important variable: time. The problem is back :-(

To make sure there is nothing wrong, I re-initialized two of my baremetal nodes (an R630 and an R730) with a super simple setup (no lag, no lacp, no lldp, no loop-detection etc), a single vlan, and not config whatsoever but the result is packet drop and delay.

Tomorrow I'll run a DPDK app to check the device's stats and put some real load on the system and see the true stability beyond ping.

Last week when I opened the ICX I immediately noticed that the main CPU was wayyyy hotter than the reported 50C. Would that be something to look into?

I can almost promise your switch is fine, it sounds like another device is flooding the switch with what end up being CPU bound packets, something like a broadcast storm etc. the fact the issue/latency went away when you unplugged all your hosts also points to this. When it happens and you can reproduce it, unplug one host/device at a time until the issue goes away, then you know which one it was. also run "show cpu" a few times to see what usage is at (although it's not super reliable in my experience)

 

[fohdeesha]

Hi all,
Can anyone confirm whether or not plugging an Brocade ICX7400-4X10GF 4x10GbE module in the back of the 7450 switch just shows up as a usable interface 1/3/1, 1/3/2 or 1/4/1, 1/4/2 etc?

yes those 4x SFP+ modules just show up like four native 10gbE ports

 

[fohdeesha]

Is it possible to configure interface management 1 with its own routing table on a ICX 6610, playing with VRF but looks like that int management 1 wont work, no VRF Forwarding options?

I'm looking at possibly of having a backdoor in case of emergency, workaround is using a jumpbox in that mgmt subnet... but not clean.

ICX 6610
Primary FCXR08030u.bin
SW: Version 08.0.30uT7f3

the ICX7xxx series supports putting the separate management port in non-default VRFs (like a management VRF), but sadly the 6 series does not. I get around this usually by creating a management VRF like usual, designating it the management vrf, making it like vlan 1500 or something, and putting 1 ethernet port in it (regular ethernet port). that eth port is now a dedicated management port in its own isolated management VRF

 

[dreamkass]

the ICX7xxx series supports putting the separate management port in non-default VRFs (like a management VRF), but sadly the 6 series does not. I get around this usually by creating a management VRF like usual, designating it the management vrf, making it like vlan 1500 or something, and putting 1 ethernet port in it (regular ethernet port). that eth port is now a dedicated management port in its own isolated management VRF

Thanks @fohdeesha also found it reading the documentation but was 8.0.9x, maybe next upgrade ICX 7xxx or Arista

 

[eduncan911]

So i'm starting to understand the power of 10G at my fingertips with these switches...

Question: Would it acceptable to create a bonded LAG pair of 10G links tagged with two VLANs, over leaving each port untagged for a specific VLAN?

I have two new servers I'm setting up and they each have dual-10G, along with various 1G ports. Running Proxmox, I'll have one 10G dedicated to server CLRNET traffic (library software for various Windows RDCs), and the other 10G dedicated to Ceph data sync on the backend.

Considering redundancy, possible failures (and the fact that I don't visit the school often), I'm now thinking of setting up a LAG group for these two 10G ports and tagging the group with the same two VLANs I was going to assign to each port untagged anyways.

The idea is to operate at 20Gbps and if one 10G link drops out for one of various reasons, the other 10G link will keep chugging along as I get an alert.

These machines won't be able to saturate a single 10G link by any means (though one could get up to 6G peak though). So I don't have to worry about one VLAN taking over the entire 20G bandwidth. Though, that would be interesting to know how to limit VLAN bandwidth on a LAG group.

Is this a good idea?

 

[LodeRunner]

Pretty sure you do not modify VLANs on a per port basis in a LAG, you add the LAG to the VLANs.

Edit: or rather, when you tag/untag a LAG to a VLAN, it does the same to the ports. I'm pretty sure you'll get an error if you try to fuss with the individual ports.

 

[richtj99]

Hi,

So I am having a strange issue. I have a Sonicwall router going to a unifi switch, going to three 6450 switches.

Its all working "sort of". The Sonicwall does most of the DHCP on the other vlans except for vlan 168 which has a windows server doing DHCP.

My camera server (on Cisco switch that I want to remove) can't hit anything plugged the brocade.

I stumbled on something with stp and thought that might be related. I enabled stp on vlan 168 and that seemed to help as offline cameras on other switches (same vlan) showed up.

I think my Cisco's had stp by default and I have a number of related issues. Can I turn stp on globally per switch vs vlan?

At one point I was messing around and put a Sonicwall port with my tagged vlans and a similar port from the unifi switch to the same brocade, different port but same tagged vlans and my kids started crying about the network. Unplugging the feeds fixed it so I think that's the issue.

 

[richtj99]

So for me I think it was a few things:

1. Moving all ports off vlan 1
2. adding a ve interface on vlan 168
3. Turning on spanning-tree 802-1w on each vlan
4. setting untagged ports vs leaving some ports without any untagged vlan

 

[infoMatt]

Pretty sure you do not modify VLANs on a per port basis in a LAG, you add the LAG to the VLANs.

Edit: or rather, when you tag/untag a LAG to a VLAN, it does the same to the ports. I'm pretty sure you'll get an error if you try to fuss with the individual ports.

IIRC the switch itself doesn't allow you to edit a single interface of a LAG. To apply VLANs or other settings to a LAG you have to apply those on the "master" or primary interface of the bond.

 

[pubsub]

Anyone know what the screw size is for the 6610 lid? I had a couple shear off on my while loosening them. I was able to unscrew the stub with some pliers but would like to replace them. They look to be M3's of some size (maybe 3-5mm length?). Not sure about the thread pitch.

 

[Rand__]

Quick question, have not found it on a search, are the modules on the 7400 line hot plug capable?

I.e can I simply rip out a 40G one and plug a 4x10 in it (or whatever) or do I need to power the switch off for that?

The FAQ for this modell only speaks about hot swappable fans/psus...

 

[eduncan911]

IIRC the switch itself doesn't allow you to edit a single interface of a LAG. To apply VLANs or other settings to a LAG you have to apply those on the "master" or primary interface of the bond.

Yep, that's what I've read as well - apply VLANs to the primary interface of the bond.

 

[richtj99]

Is there a way to do a show command to check for stp related issues?

While I thought that spanning-tree 802-1w would fix it, im doing something wrong.

 

[Boosted]

Im looking at the ICX-6610 in the 48 port POE+ flavor. One thing thats getting me hung up is licenses. There is Advanced and Premium licenses. I noticed in the datasheet there is a Premium to Advanced upgrade so that tells me that Premium < Advanced but i cannot find a breakdown of what each license gives you vs what no license gives you. Im also wondering if any of the licenses have the 10G SFP+ license baked in.

 

[itronin]

Im looking at the ICX-6610 in the 48 port POE+ flavor...
Im also wondering if any of the licenses have the 10G SFP+ license baked in.

is your usage for a business or is it home lab/hobbyist?

if home lab/hobbyist then please read the first post carefully Especially at the very top, Note 1 and Note 2... The very kind (though sometimes funny and snarky OP extraordinaire) has no/charge licenses to unlock all the features available for the ICX 6610 for home lab/hobbyist usage. You simply have to ask.

If business usage then somewhere in this thread is a discussion of the various differences between the PE and Advanced licenses...

 

[klui]

There is Advanced and Premium licenses. I noticed in the datasheet there is a Premium to Advanced upgrade so that tells me that Premium < Advanced but i cannot find a breakdown of what each license gives you vs what no license gives you. Im also wondering if any of the licenses have the 10G SFP+ license baked in.

Your query will be answered if you use the Search feature, search for "advanced premium", select This Thread, and By this thread's creator. Look at the first several posts.