Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[bitbckt]

There are no physical ports in ve1 which are Up - the ve won’t come up until at least one does.

 

[hmw]

I will make sure to report.

Damn this ICX7650 is LOUD. My Apple watch shows 63dB ~ 70 dB when near the switch. It was new in box but came with two Rev A PSUs ! On the other hand, I have like 3 Rev C and one Rev B RPS-16 PSU so it's not a problem.

But the noise ! I do need to soundproof my rack - thinking of getting some MLV (mass loaded vinyl) and putting in on the rack insides. The problem is that it is heavy - 20 sq ft of these would weight 40 lbs

@fohdeesha - the switch came with FastIron 08.0.90j - would you recommend upgrading to 10.0.0 ? Other than dropping switch & router images, is there anything worthwhile in 10.0.0? or conversely - is anything else (for example the honor based licensing) dropped in 10.0.0?

 

[NablaSquaredG]

It was new in box but came with two Rev A PSUs !

Weird... I assumed that Rev B and Rev C superseded Rev A and no Rev A are produced anymore. Can't imagine that they're shipping new ICX7650 with years old Rev A PSUs

 

[itronin]

Damn this ICX7650 is LOUD. My Apple watch shows 63dB ~ 70 dB when near the switch. It was new in box but came with two Rev A PSUs ! On the other hand, I have like 3 Rev C and one Rev B RPS-16 PSU so it's not a problem.

weird mine doesn't sound any louder than my icx6610-48p. I should check what psu's are in mine.

 

[custom90gt]

weird mine doesn't sound any louder than my icx6610-48p. I should check what psu's are in mine.

Yeah I'd love to hear back. I am pondering buying one, but I don't want something that's louder than my 6610, that's already on the edge of being too loud for my liking.

 

[hmw]

weird mine doesn't sound any louder than my icx6610-48p. I should check what psu's are in mine.

Would it be possible to take readings for both switches with an iPhone and a SPL app like Decibel X for example? I'll try doing the same ...

Also appreciate if you can check the PSUs - are you using dual PSUs or just one PSU?

 

[u4096]

I'll check that out now. The default VLAN is 1, however, I'd prefer it to be 50, since that's my physical hardware management subnet. Since the AP's management is on VLAN 1, I keep reading that its native on all ports automatically, and I can't set VLAN 1 as untagged to any interfaces. I suppose if I re-IP my APs to be on VLAN 50, it should work... That said, as shown below, the link for ve 1 is showing as Down, and I can't figure out how to get that to change it's link state to Up.

VII-SW48P01(config)#show vlan
Total PORT-VLAN entries: 7
Maximum PORT-VLAN entries: 64


[CODE]VII-SW48P01(config)#sh int brief wide

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Down    None    None None  None  Yes N/A  0   748e.f8d3.c340  Wireless AP First Floor
1/1/2      Up      Forward Full 1G    None  Yes N/A  0   748e.f8d3.c341  Wireless AP Second Floor
1/1/3      Down    None    None None  None  No  1    0   748e.f8d3.c340
ve50       Up      N/A     N/A  N/A   None  N/A N/A  N/A 748e.f8d3.c340

It's odd that your AP ports are showing "N/A" for PVID.

But I think these issues may be because the Brocade is running the router firmware. What is supposed to be doing the routing? Do you have any ACLs in this topology? If so, what switch is doing that? If it's the Ubiquiti (you said they were cores earlier), then I would run the switch firmware on the Brocade. You don't want "ve" interfaces on both as that will cause issues with routing, oddly enough even if you have "ve 50" on the Ubiquiti and "ve 50" on the Brocade. Ultimately, if VLAN 50 is to be the management VLAN, you'd be looking at implementing something like:

Ubiquiti:

• Management IP on VLAN 50 ve
• Trunk ports carry all needed VLANs tagged to Brocade (including VLAN 50)

Brocade:

• Trunk ports carry all VLANs tagged (including VLAN 50)
• The interesting nuance here is that, when you run the switch firmware, you can only run "ip address x.x.x.x/x" on the default VLAN to manage it. You won't be able to do something like "int ve 50" followed by "ip address x.x.x.x/x" on the ve anymore. It will disappear from the menu, which will simplify things. However, make the Brocade management IP be in the VLAN 50 range despite the trunk port showing a PVID of 1.
• Add an "ip default route <Ubiquiti switch>" statement on the Brocade and now all VLAN traffic will be delegated to Ubiquiti for routing.
• You shouldn't have to create ACLs on the Brocade either since routing belongs with Ubiquiti.

Now you can worry about APs being on a VLAN 50 untagged port with dual mode running and carrying the tagged traffic of the other VLANs.

Maybe others that are more seasoned at this can chime in and double check/correct my logic.

 

[eptesicus]

It's odd that your AP ports are showing "N/A" for PVID.

But I think these issues may be because the Brocade is running the router firmware. What is supposed to be doing the routing? Do you have any ACLs in this topology? If so, what switch is doing that? If it's the Ubiquiti (you said they were cores earlier), then I would run the switch firmware on the Brocade. You don't want "ve" interfaces on both as that will cause issues with routing, oddly enough even if you have "ve 50" on the Ubiquiti and "ve 50" on the Brocade. Ultimately, if VLAN 50 is to be the management VLAN, you'd be looking at implementing something like:

Ubiquiti:

• Management IP on VLAN 50 ve
• Trunk ports carry all needed VLANs tagged to Brocade (including VLAN 50)

Brocade:

• Trunk ports carry all VLANs tagged (including VLAN 50)
• The interesting nuance here is that, when you run the switch firmware, you can only run "ip address x.x.x.x/x" on the default VLAN to manage it. You won't be able to do something like "int ve 50" followed by "ip address x.x.x.x/x" on the ve anymore. It will disappear from the menu, which will simplify things. However, make the Brocade management IP be in the VLAN 50 range despite the trunk port showing a PVID of 1.
• Add an "ip default route <Ubiquiti switch>" statement on the Brocade and now all VLAN traffic will be delegated to Ubiquiti for routing.
• You shouldn't have to create ACLs on the Brocade either since routing belongs with Ubiquiti.

Now you can worry about APs being on a VLAN 50 untagged port with dual mode running and carrying the tagged traffic of the other VLANs.

Maybe others that are more seasoned at this can chime in and double check/correct my logic.

Well TIL... I had no idea there was a difference in that firmware I don't recall seeing anything in the Fohdeesha docs about that. I have pfsense handling routing, nothing doing that/ACLs on the Brocade. I'll offload some layer 3 to the Aristas once I get those up.

I'm gonna get my laptop down to my rack and mess with the config based on your recommendation. I'll report back.

Edit: Maybe I'm an idiot, but how can I get a management IP set to the switch to be accessible via any of the connected interfaces and not limit it to the MGMT port?

 

[u4096]

Well TIL... I had no idea there was a difference in that firmware I don't recall seeing anything in the Fohdeesha docs about that. I have pfsense handling routing, nothing doing that/ACLs on the Brocade. I'll offload some layer 3 to the Aristas once I get those up.

Yes, the router firmware has an R in it-> SPR08030x where R = Router. The switch one has an S in the same place. SPS08030x

To see what images you have already in flash you can do:

> show flash

and it should look something like

Compressed Pri Code ..<stuff here>.. (SPS08030e.bin)
Compressed Sec Code ..<stuff here.. (SPS08030e.bin)

If by any chance, the secondary image is SPR just run:

> boot flash secondary

Edit: Maybe I'm an idiot, but how can I get a management IP set to the switch to be accessible via any of the connected interfaces and not limit it to the MGMT port?

Once you get the switch firmware on it, just run:

> conf t
> ip address x.x.x.x/24

That will be the management IP reachable via any of the ports with a default PVID. You just have to assign your laptop a static IP in the management range since the DHCP server won't be active by default.

 

[bitbckt]

I don't recall seeing anything in the Fohdeesha docs about that.

It‘s not worth mentioning because approx. no one uses switch only firmware. The router firmware contains all the same functionality, plus L3 features if/when they’re necessary later.

 

[custom90gt]

weird mine doesn't sound any louder than my icx6610-48p. I should check what psu's are in mine.

Any update on this? Sorry I would love to buy one, but if I do and it's crazy loud I think my wife will be less than pleased...

*on edit* - it looks like these use the same power supplies as the 6610. I have two revision C power supplies already.

 

[NablaSquaredG]

If it's the Ubiquiti (you said they were cores earlier), then I would run the switch firmware on the Brocade.

Please don't do that. Thanks.

 

[itronin]

Any update on this? Sorry I would love to buy one, but if I do and it's crazy loud I think my wife will be less than pleased...

*on edit* - it looks like these use the same power supplies as the 6610. I have two revision C power supplies already.

sorry work has been really busy - hoping to shut it down, pull the psus on Friday, and maybe get some comparable sound readings of the 7650 and 6610. re-iterating. audibly I cannot tell the difference between the 7650 and 6610 - but everyone's ear's are different.

and yeah, IIRC they are the same psu's. I have two B's in my 6610 and 2 fans. my7650 has two psu's and 2 fans + the front 4x10gbe module installed but I'm not using.

 

[Blue)(Fusion]

It‘s not worth mentioning because approx. no one uses switch only firmware. The router firmware contains all the same functionality, plus L3 features if/when they’re necessary later.

FWIW, I use the switch firmware on my 6450 since my 6610 does router duty. The switch-only firmware works with IPv6 better when the prefixes may change (i,e, home network). The switch-only firmware will auto configure IPv6 advertised prefixes with SLAAC like any other end devices. The router firmware can take a prefix and add the eiu-64 address or be configured manually but not SLAAC is not an option.

Honestly for anyone using the ICX switches for just L2 switches and not using their L3 abilities, go with the switch firmware and prevent further odd routing issues that you may cause without intention.

 

[CHANABRA]

so i have followed a couple peoples recommendations for fans for 7250-24P, all of them fall very short... maybe I am too conservative i just dont like +15C and the poor fellow whos switch psu blew got me being careful as well summer is coming and im not risking it..
anyhow, I tested the Arctic S4028-15K... it doesn't trip the unit as the rpm is high enough, they are very easy to use (work in 2, 3 or 4 sockets), 19.22 CFM vs 24 CFM with the OEM Foxconn, so not too far off... and they have a very powerful airflow at 6v (setting 1).

I need to find my sound meter and I can give a accurate comparison between the two but with the ear test it is signficantly quieter. I also recently took an unused 12v Xa adapter and fed them into wago connectors then through where the 3rd fan on the 48P's screw holes, and have that powering a 60x60x10 brushless fan over the heatsink and am making use of the failed fans I bought recommended on this thread (one at least) and its blowing across diagonally from the corner on the 24P where the RJ45 ports stop... am going to pay very close attention to it and report back if temps get uncomfortable and i am very conservative i dont want extremely elvatedf temps..

so far a few hours in

edit: power 2 R710's LACP, and 1 poe camera, and only 5-7 other rj45 + 4/8 SFP+

 

[Blue)(Fusion]

I'll check that out now. The default VLAN is 1, however, I'd prefer it to be 50, since that's my physical hardware management subnet. Since the AP's management is on VLAN 1, I keep reading that its native on all ports automatically, and I can't set VLAN 1 as untagged to any interfaces. I suppose if I re-IP my APs to be on VLAN 50, it should work... That said, as shown below, the link for ve 1 is showing as Down, and I can't figure out how to get that to change it's link state to Up.

VII-SW48P01(config)#show vlan
Total PORT-VLAN entries: 7
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1)   3   4   5   6  11  12  17  18  19  20  37  38
Untagged Ports: (U1/M1)  39  40  41  42  43  44  45  46  47  48
Untagged Ports: (U1/M2)   3   4
   Tagged Ports: None
   Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled
  
PORT-VLAN 50, Name v50_MGMTPhysical, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1)   7   8   9  10  13  14  15  16
   Tagged Ports: (U1/M2)   1   2
   Uplink Ports: (U1/M2)   1   2
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled
  
PORT-VLAN 60, Name v60_MGMTUser, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1)  31  32  33  34  35  36
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2
   Uplink Ports: (U1/M2)   1   2
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled
  
PORT-VLAN 100, Name v100_Guest, Priority level0, Spanning tree Off
Untagged Ports: None
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2
   Uplink Ports: (U1/M2)   1   2
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled
  
PORT-VLAN 110, Name v110_Wiferino, Priority level0, Spanning tree Off
Untagged Ports: None
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2
   Uplink Ports: (U1/M2)   1   2
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled
  
PORT-VLAN 120, Name v120_IOT, Priority level0, Spanning tree Off
Untagged Ports: None
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2
   Uplink Ports: (U1/M2)   1   2
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled
  
PORT-VLAN 130, Name v130_Cams, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1)  21  22  23  24  25  26  27  28  29  30
   Tagged Ports: (U1/M1)   1   2
   Tagged Ports: (U1/M2)   1   2
   Uplink Ports: (U1/M2)   1   2
DualMode Ports: None
Mac-Vlan Ports: None
     Monitoring: Disabled

VII-SW48P01(config)#sh int brief wide

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Down    None    None None  None  Yes N/A  0   748e.f8d3.c340  Wireless AP First Floor
1/1/2      Up      Forward Full 1G    None  Yes N/A  0   748e.f8d3.c341  Wireless AP Second Floor
1/1/3      Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/4      Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/5      Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/6      Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/7      Down    None    None None  None  No  50   0   748e.f8d3.c340  MX7000 MGMT
1/1/8      Down    None    None None  None  No  50   0   748e.f8d3.c340  MX7000 MGMT
1/1/9      Up      Forward Full 1G    None  No  50   0   748e.f8d3.c340  NAS01 IPMI
1/1/10     Up      Forward Full 1G    None  No  50   0   748e.f8d3.c340  NAS02 IPMI
1/1/11     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/12     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/13     Down    None    None None  None  No  50   0   748e.f8d3.c340  APC UPS MGMT
1/1/14     Down    None    None None  None  No  50   0   748e.f8d3.c340
1/1/15     Down    None    None None  None  No  50   0   748e.f8d3.c340  APC PDU 1
1/1/16     Down    None    None None  None  No  50   0   748e.f8d3.c340  APC PDU 2
1/1/17     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/18     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/19     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/20     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/21     Up      Forward Full 100M  None  No  130  0   748e.f8d3.c354  Surveillance Camera
1/1/22     Up      Forward Full 100M  None  No  130  0   748e.f8d3.c355  Surveillance Camera
1/1/23     Up      Forward Full 100M  None  No  130  0   748e.f8d3.c356  Surveillance Camera
1/1/24     Up      Forward Full 100M  None  No  130  0   748e.f8d3.c357  Surveillance Camera
1/1/25     Up      Forward Full 100M  None  No  130  0   748e.f8d3.c358  Surveillance Camera
1/1/26     Up      Forward Full 100M  None  No  130  0   748e.f8d3.c359  Surveillance Camera
1/1/27     Down    None    None None  None  No  130  0   748e.f8d3.c35a  Surveillance Camera
1/1/28     Down    None    None None  None  No  130  0   748e.f8d3.c35b  Surveillance Camera
1/1/29     Down    None    None None  None  No  130  0   748e.f8d3.c35c  Surveillance Camera
1/1/30     Down    None    None None  None  No  130  0   748e.f8d3.c35d  Surveillance Camera
1/1/31     Down    None    None None  None  No  60   0   748e.f8d3.c35e  Mike's Office
1/1/32     Down    None    None None  None  No  60   0   748e.f8d3.c35f  Mike's Office
1/1/33     Down    None    None None  None  No  60   0   748e.f8d3.c360  Mike's Office
1/1/34     Down    None    None None  None  No  60   0   748e.f8d3.c361  Mike's Office
1/1/35     Down    None    None None  None  No  60   0   748e.f8d3.c362  Mike's Office
1/1/36     Down    None    None None  None  No  60   0   748e.f8d3.c363  Mike's Office
1/1/37     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/38     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/39     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/40     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/41     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/42     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/43     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/44     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/45     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/46     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/47     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/1/48     Down    None    None None  None  No  1    0   748e.f8d3.c340
1/2/1      Up      Forward Full 10G   None  Yes N/A  0   748e.f8d3.c340  Uplink to Core
1/2/2      Disable None    None None  None  Yes N/A  0   748e.f8d3.c340  Uplink to Core
1/2/3      Down    None    None None  None  No  1    0   748e.f8d3.c340
1/2/4      Down    None    None None  None  No  1    0   748e.f8d3.c340
mgmt1      Down    None    None None  None  No  None 0   748e.f8d3.c340
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Down    N/A     N/A  N/A   None  N/A N/A  N/A 748e.f8d3.c340
ve50       Up      N/A     N/A  N/A   None  N/A N/A  N/A 748e.f8d3.c340

It's odd that your AP ports are showing "N/A" for PVID.

But I think these issues may be because the Brocade is running the router firmware. What is supposed to be doing the routing? Do you have any ACLs in this topology? If so, what switch is doing that? If it's the Ubiquiti (you said they were cores earlier), then I would run the switch firmware on the Brocade. You don't want "ve" interfaces on both as that will cause issues with routing, oddly enough even if you have "ve 50" on the Ubiquiti and "ve 50" on the Brocade. Ultimately, if VLAN 50 is to be the management VLAN, you'd be looking at implementing something like:

Ubiquiti:

• Management IP on VLAN 50 ve
• Trunk ports carry all needed VLANs tagged to Brocade (including VLAN 50)

Brocade:

• Trunk ports carry all VLANs tagged (including VLAN 50)
• The interesting nuance here is that, when you run the switch firmware, you can only run "ip address x.x.x.x/x" on the default VLAN to manage it. You won't be able to do something like "int ve 50" followed by "ip address x.x.x.x/x" on the ve anymore. It will disappear from the menu, which will simplify things. However, make the Brocade management IP be in the VLAN 50 range despite the trunk port showing a PVID of 1.
• Add an "ip default route <Ubiquiti switch>" statement on the Brocade and now all VLAN traffic will be delegated to Ubiquiti for routing.
• You shouldn't have to create ACLs on the Brocade either since routing belongs with Ubiquiti.

Now you can worry about APs being on a VLAN 50 untagged port with dual mode running and carrying the tagged traffic of the other VLANs.

Maybe others that are more seasoned at this can chime in and double check/correct my logic.

Sounds to me like you need to tell your switch which VLAN is used for management traffic.

Also, if you are in switch-only firmware (which is fine despite what others may tell you, if you are not using or going to use L3 functionality), do this:

vlan 50
management-vlan
default-gateway 10.1.1.1 1
exit
ip address 10.1.1.2

Replace the IP address and gateway address as appropriate. Take not there is a "1" after the gateway address as you can have more than one gateway.

 

[zunder1990]

It‘s not worth mentioning because approx. no one uses switch only firmware. The router firmware contains all the same functionality, plus L3 features if/when they’re necessary later.

that is not true, I have 8x of the 6610 and only use the switch firmware, I prefer this version

 

[bitbckt]

that is not true, I have 8x of the 6610 and only use the switch firmware, I prefer this version

I did say “approx.”, however there is good reason for Ruckus merging the two into a single (router!) image in 10.x.

As you’re still using 6610s, that probably won’t affect you until approx. the year 2050.

 

[dswartz]

FWIW, I use the switch firmware on my 6450 since my 6610 does router duty. The switch-only firmware works with IPv6 better when the prefixes may change (i,e, home network). The switch-only firmware will auto configure IPv6 advertised prefixes with SLAAC like any other end devices. The router firmware can take a prefix and add the eiu-64 address or be configured manually but not SLAAC is not an option.

Honestly for anyone using the ICX switches for just L2 switches and not using their L3 abilities, go with the switch firmware and prevent further odd routing issues that you may cause without intention.

Yeah, I don't get the hate for L2 firmware. I've got 2 fortigate firewalls that do all the routing I need in my home lab.

 

[hmw]

Any update on this? Sorry I would love to buy one, but if I do and it's crazy loud I think my wife will be less than pleased...

*on edit* - it looks like these use the same power supplies as the 6610. I have two revision C power supplies already.

It isn't the power supplies. The ICX7650 has different fans and it runs the PSU harder hence sounding harsher and louder than the ICX6610

I measured SPL for both the switches inside my enclosed 42U rack, doors closed and standing around 5 feet away from the rack

Using Decibel X on the iPhone. This is reference + background



Switching the ICX 6610 on



After a minute or two the ICX6610 ramps the fans down



Now switching the ICX7650 on - note the different harmonics



The 7650 takes somewhat longer to ramp the fans down than the 6610 - and also has different and more unpleasant sounding harmonics

and

My rack is in the basement but at night when the house is quiet, you can hear the 7650 - but the 6610 I think has a lower frequency and blends into the background noise very nicely

Since those frequency peaks are annoying - I've ordered some MLV for the rack - will try to put acoustic foam and MLV so that I can dampen those frequencies.

The 7650 pulls a quite a bit more power - it consumes 120W min at idle, the 6610 does 80 ~ 85W at idle