Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Nikotine

Member
Mar 17, 2021
35
0
6
I don't have these specific passive copper direct attached cables. DACs are different from transceiver + fiber cables. You won't be able to get temperature, power readings, etc. from DACs.
It would be to connect the switch (6450) to a NAS over a short distance, so not sure if fiber is needed.
My understanding is that copper is fine for short distances.
The options are very confusing.

Looking for 57-0000075-01, I find these for €15:
Or the same for €238:

This doesn't make any sense to me...
 

m4r1k

Member
Nov 4, 2016
75
8
8
35
Sorry I missed that you had the running-config linked already.

Th pasted ping shows the packet delays when pinging the switch.
Hey there!

Apologies for the late answer, in such shape my lab didn't really work, and setting something custom up takes time.

tl;dr is weird without ANY change, this morning re-applying my config (same as the one on GitHub) the issue is essentially gone (well, I of course upgraded to the latest firmware but two days ago the situation was unable)

I also created a much simpler setup: two 10Gbps interfaces of two different R630 in the same vlan, ping between the two and no drop nor delay whatsoever.

Should I take that the switch is about to die?

In your latest post, you say it happens to any traffic going through the switch (i.e. edge device to edge device). If the latter is true, than there may be an issue. If pings between edge devices are not experiencing this issue, than it is likely normal. Traffic sent to the switch itself is shunted to the management CPU and given low priority. The pings you see are higher than I've seen thus far on my setup, however I'm not sure how loaded up your switch is with traffic.
See below the system logs
Code:
#show logging
Syslog logging: enabled ( 0 messages dropped, 0 flushes, 0 overruns)
    Buffer logging: level ACDMEINW, 58 messages logged
    level code: A=alert C=critical D=debugging M=emergency E=error
                I=informational N=notification W=warning

Static Log Buffer:
Apr  5 13:55:24:I:System: Stack unit 1   Power supply 1  is up
Apr  5 13:55:24:I:System: Stack unit 1   Power supply 2  is up

Dynamic Log Buffer (1000 lines):
Apr  5 13:58:51:I:Security: SSH login by un-authenticated SSH user from src IP 192.168.178.68 from src MAC 3023.03e2.2a39 to PRIVILEGED EXEC mode using RSA as Server Host Key.
Apr  5 13:58:50:I:Security: SSH login by un-authenticated SSH user from src IP 192.168.178.68 from src MAC 3023.03e2.2a39 to USER EXEC mode using RSA as Server Host Key.
Apr  5 13:57:44:I:NTP: System clock is synchronized to 82.161.139.11.
Apr  5 13:56:35:I:Security: Time is updated by NTP server "82.161.139.11" from  "01:00:00.000 GMT+01 Mon Jan 01 1900 " to "13:56:35.499 GMT+01 Mon Apr 05 2021 "
Apr  5 13:55:30:I:System: Interface ethernet 1/1/5, state up
Apr  5 13:55:27:I:System: Interface ethernet 1/1/5, state down
Apr  5 13:55:27:I:System: Interface ethernet 1/1/14, state up
Apr  5 13:55:27:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is force-up.
Apr  5 13:55:27:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is up.
Apr  5 13:55:27:I:System: Interface ethernet 1/1/16, state up
Apr  5 13:55:26:I:System: Interface ethernet 1/1/13, state up
Apr  5 13:55:26:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is force-up.
Apr  5 13:55:26:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is up.
Apr  5 13:55:26:I:System: Interface ethernet 1/1/15, state up
Apr  5 13:55:26:I:System: Interface ethernet 1/1/5, state up
Apr  5 13:55:26:I:System: Logical link on dynamic lag interface ethernet 1/3/8 is up.
Apr  5 13:55:26:I:System: Interface ethernet 1/3/8, state up
Apr  5 13:55:26:I:System: Logical link on dynamic lag interface ethernet 1/3/7 is up.
Apr  5 13:55:26:I:System: Interface ve 140, state up
Apr  5 13:55:26:I:System: Interface ve 130, state up
Apr  5 13:55:26:I:System: Interface ve 120, state up
Apr  5 13:55:26:I:System: Interface ve 110, state up
Apr  5 13:55:26:I:System: Interface ve 100, state up
Apr  5 13:55:25:I:Trunk: Group (1/3/7, 1/3/8) created by 802.3ad link-aggregation module.
Apr  5 13:55:25:I:System: dynamic lag 30, has new peer info (priority=65535,id=e443.4b44.5b2e,key=15) (N/A)
Apr  5 13:55:25:I:System: Interface ethernet 1/1/7, state up
Apr  5 13:55:25:I:System: Interface ethernet 1/1/4, state up
Apr  5 13:55:25:I:System: Interface ethernet 1/1/6, state up
Apr  5 13:55:25:I:System: Interface ethernet 1/1/2, state up
Apr  5 13:55:24:I:System: Stack unit 1   Power supply 2  is up
Apr  5 13:55:24:I:System: Stack unit 1   Power supply 1  is up
Apr  5 13:55:24:I:System: Interface ethernet 1/1/1, state up
Apr  5 13:55:24:I:System: Interface ve 178, state up
Apr  5 13:55:23:I:System: Logical link on force-up dynamic lag interface ethernet 1/3/7 is back to LACP control.
Apr  5 13:55:23:I:System: Interface ve 178, state down
Apr  5 13:55:23:I:System: Interface ve 140, state down
Apr  5 13:55:23:I:System: Interface ve 130, state down
Apr  5 13:55:23:I:System: Interface ve 120, state down
Apr  5 13:55:23:I:System: Interface ve 110, state down
Apr  5 13:55:23:I:System: Interface ve 100, state down
Apr  5 13:55:23:I:System: Logical link on dynamic lag interface ethernet 1/3/7 is force-up.
Apr  5 13:55:23:I:System: Logical link on dynamic lag interface ethernet 1/3/7 is up.
Apr  5 13:55:23:I:System: Interface ethernet 1/3/7, state up
Apr  5 13:55:23:I:System: Interface ve 178, state up
Apr  5 13:55:23:I:System: Interface ve 140, state up
Apr  5 13:55:23:I:System: Interface ve 130, state up
Apr  5 13:55:23:I:System: Interface ve 120, state up
Apr  5 13:55:23:I:System: Interface ve 110, state up
Apr  5 13:55:23:I:System: Interface ve 100, state up
Apr  5 13:55:23:I:System: Warm start
Apr  5 13:55:00:I:System: Port init success Stack unit 1 Port 1/2/1 Lane 0 T 0 R 0 Type 0:  00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x000
Apr  5 13:52:28:I:DHCPC: protocol disabled by user
Apr  5 13:52:28:I:NTP: client association is mobilized for 129.250.35.250.
Apr  5 13:52:28:I:NTP: client association is mobilized for 82.161.139.11.
Apr  5 13:52:28:I:NTP: client association is mobilized for 213.109.127.82.
Apr  5 13:52:28:I:NTP: The system clock is not synchronized to any time source.
Apr  5 13:52:28:I:NTP: client association is mobilized for 95.211.160.148.
Apr  5 13:52:28:I:NTP: The system clock is not synchronized and does not have a reference configured.
 

DASHIP

New Member
May 4, 2016
15
0
1
54
Be careful making sure you check the model number before buying. The model numbering scheme is confusing. I just purchased a 6610-48-PE thinking it was a 6610-48P-E. The latter has PoE ports, while the former does not. If you are unsure, be sure to check the datasheets listed on the first page of this thread. Here is an excerpt for the 6610-48 as an example. Notice how close some of the model numbers are. With the noted inaccuracy of eBay postings, it is easy to get the wrong switch. Also, the models with PoE have a "P" at the end of the model name on the front of the switch, in white lettering: "ICX 6610-48P". If the "P" is not present on the front, it is not a PoE model.
 

Attachments

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
tl;dr is weird without ANY change, this morning re-applying my config (same as the one on GitHub) the issue is essentially gone (well, I of course upgraded to the latest firmware but two days ago the situation was unable)
I suspect the issue is and never was with the switch but either a bad connection/cable somewhere or a routing table on an edge device doing funky things. Be sure to check for packet errors on your end device(s).

I also created a much simpler setup: two 10Gbps interfaces of two different R630 in the same vlan, ping between the two and no drop nor delay whatsoever.
This is what really matters.


Should I take that the switch is about to die?
Certainly not. Traffic sent to the switch management interface (i.e. ICMP ping) is removed from the normal path through the ASICs and sent to the management CPU. It is also fairly low priority, so if the management CPU is doing something else, the ICMP pings/SNMP returns will end up with varied latency. Meanwhile, traffic passing through the switch normally from device to device will experience none of the latency that the management CPU is returning in ping results.
 

m4r1k

Member
Nov 4, 2016
75
8
8
35
I suspect the issue is and never was with the switch but either a bad connection/cable somewhere or a routing table on an edge device doing funky things. Be sure to check for packet errors on your end device(s).


This is what really matters.


Certainly not. Traffic sent to the switch management interface (i.e. ICMP ping) is removed from the normal path through the ASICs and sent to the management CPU. It is also fairly low priority, so if the management CPU is doing something else, the ICMP pings/SNMP returns will end up with varied latency. Meanwhile, traffic passing through the switch normally from device to device will experience none of the latency that the management CPU is returning in ping results.
I think we're speaking too early without an important variable: time. The problem is back :-(

To make sure there is nothing wrong, I re-initialized two of my baremetal nodes (an R630 and an R730) with a super simple setup (no lag, no lacp, no lldp, no loop-detection etc), a single vlan, and not config whatsoever but the result is packet drop and delay.

Tomorrow I'll run a DPDK app to check the device's stats and put some real load on the system and see the true stability beyond ping.

Last week when I opened the ICX I immediately noticed that the main CPU was wayyyy hotter than the reported 50C. Would that be something to look into?
 

dennisp

New Member
Apr 1, 2021
18
13
3
I'm 94 pages in and already ordered a pair of 6610 to replace the Cisco 3750 I use at home. Also I get to learn some new tech and hopefully save some power in the process. 100+ pages to go but I skipped ahead to say thanks to @fohdeesha for sharing all of this info.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
I think we're speaking too early without an important variable: time. The problem is back :-(

To make sure there is nothing wrong, I re-initialized two of my baremetal nodes (an R630 and an R730) with a super simple setup (no lag, no lacp, no lldp, no loop-detection etc), a single vlan, and not config whatsoever but the result is packet drop and delay.

Tomorrow I'll run a DPDK app to check the device's stats and put some real load on the system and see the true stability beyond ping.

Last week when I opened the ICX I immediately noticed that the main CPU was wayyyy hotter than the reported 50C. Would that be something to look into?
I can almost promise your switch is fine, it sounds like another device is flooding the switch with what end up being CPU bound packets, something like a broadcast storm etc. the fact the issue/latency went away when you unplugged all your hosts also points to this. When it happens and you can reproduce it, unplug one host/device at a time until the issue goes away, then you know which one it was. also run "show cpu" a few times to see what usage is at (although it's not super reliable in my experience)
 
  • Like
Reactions: dswartz

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Is it possible to configure interface management 1 with its own routing table on a ICX 6610, playing with VRF but looks like that int management 1 wont work, no VRF Forwarding options?

I'm looking at possibly of having a backdoor in case of emergency, workaround is using a jumpbox in that mgmt subnet... but not clean.

ICX 6610
Primary FCXR08030u.bin
SW: Version 08.0.30uT7f3
the ICX7xxx series supports putting the separate management port in non-default VRFs (like a management VRF), but sadly the 6 series does not. I get around this usually by creating a management VRF like usual, designating it the management vrf, making it like vlan 1500 or something, and putting 1 ethernet port in it (regular ethernet port). that eth port is now a dedicated management port in its own isolated management VRF
 
  • Like
Reactions: tommybackeast

dreamkass

Member
Aug 14, 2012
31
5
8
the ICX7xxx series supports putting the separate management port in non-default VRFs (like a management VRF), but sadly the 6 series does not. I get around this usually by creating a management VRF like usual, designating it the management vrf, making it like vlan 1500 or something, and putting 1 ethernet port in it (regular ethernet port). that eth port is now a dedicated management port in its own isolated management VRF
Thanks @fohdeesha also found it reading the documentation but was 8.0.9x, maybe next upgrade ICX 7xxx or Arista
 

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
So i'm starting to understand the power of 10G at my fingertips with these switches... :)

Question: Would it acceptable to create a bonded LAG pair of 10G links tagged with two VLANs, over leaving each port untagged for a specific VLAN?

I have two new servers I'm setting up and they each have dual-10G, along with various 1G ports. Running Proxmox, I'll have one 10G dedicated to server CLRNET traffic (library software for various Windows RDCs), and the other 10G dedicated to Ceph data sync on the backend.

Considering redundancy, possible failures (and the fact that I don't visit the school often), I'm now thinking of setting up a LAG group for these two 10G ports and tagging the group with the same two VLANs I was going to assign to each port untagged anyways.

The idea is to operate at 20Gbps and if one 10G link drops out for one of various reasons, the other 10G link will keep chugging along as I get an alert.

These machines won't be able to saturate a single 10G link by any means (though one could get up to 6G peak though). So I don't have to worry about one VLAN taking over the entire 20G bandwidth. Though, that would be interesting to know how to limit VLAN bandwidth on a LAG group.

Is this a good idea?
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Pretty sure you do not modify VLANs on a per port basis in a LAG, you add the LAG to the VLANs.

Edit: or rather, when you tag/untag a LAG to a VLAN, it does the same to the ports. I'm pretty sure you'll get an error if you try to fuss with the individual ports.
 
  • Like
Reactions: eduncan911

richtj99

Member
Jul 8, 2017
70
1
8
50
Hi,

So I am having a strange issue. I have a Sonicwall router going to a unifi switch, going to three 6450 switches.

Its all working "sort of". The Sonicwall does most of the DHCP on the other vlans except for vlan 168 which has a windows server doing DHCP.

My camera server (on Cisco switch that I want to remove) can't hit anything plugged the brocade.

I stumbled on something with stp and thought that might be related. I enabled stp on vlan 168 and that seemed to help as offline cameras on other switches (same vlan) showed up.

I think my Cisco's had stp by default and I have a number of related issues. Can I turn stp on globally per switch vs vlan?

At one point I was messing around and put a Sonicwall port with my tagged vlans and a similar port from the unifi switch to the same brocade, different port but same tagged vlans and my kids started crying about the network. Unplugging the feeds fixed it so I think that's the issue.
 

richtj99

Member
Jul 8, 2017
70
1
8
50
So for me I think it was a few things:

1. Moving all ports off vlan 1
2. adding a ve interface on vlan 168
3. Turning on spanning-tree 802-1w on each vlan
4. setting untagged ports vs leaving some ports without any untagged vlan
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
Pretty sure you do not modify VLANs on a per port basis in a LAG, you add the LAG to the VLANs.

Edit: or rather, when you tag/untag a LAG to a VLAN, it does the same to the ports. I'm pretty sure you'll get an error if you try to fuss with the individual ports.
IIRC the switch itself doesn't allow you to edit a single interface of a LAG. To apply VLANs or other settings to a LAG you have to apply those on the "master" or primary interface of the bond.
 

pubsub

New Member
Apr 7, 2021
2
0
1
Anyone know what the screw size is for the 6610 lid? I had a couple shear off on my while loosening them. I was able to unscrew the stub with some pliers but would like to replace them. They look to be M3's of some size (maybe 3-5mm length?). Not sure about the thread pitch.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Quick question, have not found it on a search, are the modules on the 7400 line hot plug capable?

I.e can I simply rip out a 40G one and plug a 4x10 in it (or whatever) or do I need to power the switch off for that?

The FAQ for this modell only speaks about hot swappable fans/psus...
 

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
IIRC the switch itself doesn't allow you to edit a single interface of a LAG. To apply VLANs or other settings to a LAG you have to apply those on the "master" or primary interface of the bond.
Yep, that's what I've read as well - apply VLANs to the primary interface of the bond.
 

richtj99

Member
Jul 8, 2017
70
1
8
50
Is there a way to do a show command to check for stp related issues?

While I thought that spanning-tree 802-1w would fix it, im doing something wrong.
 

Boosted

New Member
Feb 5, 2021
4
0
1
Im looking at the ICX-6610 in the 48 port POE+ flavor. One thing thats getting me hung up is licenses. There is Advanced and Premium licenses. I noticed in the datasheet there is a Premium to Advanced upgrade so that tells me that Premium < Advanced but i cannot find a breakdown of what each license gives you vs what no license gives you. Im also wondering if any of the licenses have the 10G SFP+ license baked in.