Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Spearfoot · Jun 29, 2021

mmx said:
Hi all, longtime lurker, first time poster (in this thread, any way).

Thanks to @fohdeesha and the immense amount of knowledge found in this thread by numerous contributors, I bought 4x 6610s last year to replace the crappy Cisco SG200/SG220s we have at work. I've been taking my time with the migration as it's my first time working with such powerful devices and I want to make sure I've got everything covered. My goal is to offload inter-VLAN routing from pfSense onto the 6610s (via a transit VLAN).

I want to share a small discovery I made today in hopes of making these switches friendlier to newcomers like me.

One of my biggest challenges was managing ACLs through the CLI. I'm used to centrally managing firewall rules through a web panel, so having a non-visual interface takes some getting used to. Some of my concerns & questions were:

How can I add/modify/remove a single ACL entry in an access-list?

How can I reorder ACLs in an access-list?

How can I write ACLs more efficiently? Is there an equivalent to pfSense's IP/Port aliases?

This is where Brocade Network Advisor comes in. Now I know it's EOL, but it's still quite a valuable tool for someone like me as it answers all the questions above. BNA is probably nothing new to the experienced out there, yet it's barely mentioned in this thread apart from @Jason Antes bringing it up in April 2021 and last week. I think it deserves some recognition even if it's mostly archaic and superseded by Ruckus.

One of the greatest features in BNA is the fact that you can create Networks, Network Groups, Services and Service Groups. These are basically an alternative to pfSense's aliases, and they're extremely useful when writing ACLs for several domain networks. You can even include Groups in Groups, equivalent to referencing an alias inside another alias in pfSense. Just this feature alone avoids having to repeat yourself, thus avoiding mistakes when writing ACLs for dozens of networks with similar rules. I've read through the documentation provided by Ruckus, and there's no way to replicate this functionality through the CLI (as far as I can tell).

For example, I want to create a single Service Group for all Active Directory Domain Controller ports. Here's a few screenshots to showcase the process. Service ports are protocol-specific, however you will still have to create separate ACLs for TCP and UDP. The final screenshot will show you how every individual entry gets created automagically with only 1-2 entries created in BNA.

For anyone interested, the version I have found online is 14.2.12 (IP only, no SAN support) and it doesn't require a paid licence. I've deployed this particular version without issue.

I hope this helps someone out there!

Sounds interesting!

Do you have a Premium Support account at Ruckus? I get "That file is only available to Premium Support users." when I try downloading it from Ruckus. Also, the latest version I found there is 14.2.11:

Brocade Network Advisor (BNA) | Products | Ruckus Wireless Support

support.ruckuswireless.com

mmx · Jun 30, 2021

Spearfoot said:
Sounds interesting!

Do you have a Premium Support account at Ruckus? I get "That file is only available to Premium Support users." when I try downloading it from Ruckus. Also, the latest version I found there is 14.2.11:

https://support.ruckuswireless.com/...bna#sort=relevancy&f:mad:commonproducts=[BNA]

Nope, I don't have any paid support plans. Check out the link I posted to the Reddit thread; one of the comments will give you a Google Drive link to download 14.2.12.

Jason Antes · Jun 30, 2021

I have versions from 12.31 to 14.41. None of these versions like installing in Server 2019 so I went the route of installing on 2012R2 and then doing an in-place upgrade to 2019 after installation. The license I have required me to install the older 12.x series first and then upgrade to 14.x which is why I had to do the 2012R2 route. It is a great tool. PM me if you need help.

Jason Antes · Jun 30, 2021

I need to make a clarification, the last version I have from Brocade of BNA that supports IP is 14.3.0. 14.4.0 is SAN only.

Sundar · Jun 30, 2021

juey said:
You can also buy Noctua NF-A4x20 FLX fans, i have modded 3 6450-24p with these ...

@juey : I just received my ICX 6450-24P. I do not use PoE and I have disabled PoE. The stock fans are still too loud for me. I am thinking of buying the Noctua NF-A4x20 FLX fans (I have been very happy with them in other switches like the Quanta LB9a).
I have a few questions:
- Will the standard 3-pin connector on the Noctua fans __JUST_WORK__?
- Or do I need to swap around the wires in any way?
- Did you replace BOTH the stock Sunon fans, or just one?

TIA

NateS · Jul 1, 2021

Sundar said:
@juey : I just received my ICX 6450-24P. I do not use PoE and I have disabled PoE. The stock fans are still too loud for me. I am thinking of buying the Noctua NF-A4x20 FLX fans (I have been very happy with them in other switches like the Quanta LB9a).
I have a few questions:
- Will the standard 3-pin connector on the Noctua fans __JUST_WORK__?
- Or do I need to swap around the wires in any way?
- Did you replace BOTH the stock Sunon fans, or just one?

TIA

You need to swap the wires with any fan replacement -- the switch uses a non-standard pinout. It's very easy to do though. The center wire stays in place, and the left and right swap. Alternatively, you could also just break off the latching tab, then plug the whole thing in backwards.

On my switch, I used these Sunon fans for replacement. They're not perfectly silent, but very quiet, which was good enough for me. I'm sure the Noctuas would be even quieter. In my case, I swapped both, but since this switch doesn't care if they're present at all, it should be possible to just swap one and disconnect the other -- just watch you're temps if you're running a heavy load.

Jason Antes · Jul 2, 2021

Jason Antes said:
I need to make a clarification, the last version I have from Brocade of BNA that supports IP is 14.3.0. 14.4.0 is SAN only.

I am uploading 14.3 to a google drive location and will share it out when I have it uploaded.

I looked at the 14.2.12 version compared from Rukus to the 14.3 from Brocade. If your goal is to only manage IP, go with 14.2.12. It has some CVE fixes and supports a newer version of JRE. I'm going to spin it up on a test VM and see if I can install it and use my licenses on it directly on 2019 Server.

Sundar · Jul 2, 2021

NateS said:
You need to swap the wires with any fan replacement -- the switch uses a non-standard pinout. It's very easy to do though. The center wire stays in place, and the left and right swap. Alternatively, you could also just break off the latching tab, then plug the whole thing in backwards.

On my switch, I used these Sunon fans for replacement. They're not perfectly silent, but very quiet, which was good enough for me. I'm sure the Noctuas would be even quieter. In my case, I swapped both, but since this switch doesn't care if they're present at all, it should be possible to just swap one and disconnect the other -- just watch you're temps if you're running a heavy load.

Thanks, @NateS

richtj99 · Jul 3, 2021

epicurean said:
The network in my home is suddenly very slow, especially when viewing the IP cameras. I used to use wireguard on my phone and can see all my ip cameras almost instantly, but now a few cameras come on and off and very sluggish. Plex also sluggish with nvidia shield. Intermittenly network connection also goes off. I have rebooted my switches but does not seem to make a difference.
I am not sure where to start troubleshooting. A dedicated PFsense router is connected to a ICX6450 48 POE switch, which is also connectedt o a unifi 16 POE switch. Half of my IP cams are in the unifi switch (which I intend to migrate to the ICX6450), and the other half in the ICX6450. 3 x R610 ruckus APs are connected to the ICX6450 , as is a single ruckus H510 as well.
The IP cameras in the ICX6450 are on vlan 80, whilst those in the unifi switch is on the main vlan 1 - in the process of being moved to vlan 80.

Any help appreciated. much thanks!

Im wondering if this has to do with the pfsense firewall & the speed of Vlan 1 talking to vlan 80? I notice with my firewall, even with port rules enabled for inter-vlan traffic, there is a speed difference in transferring files from vlan 20 (nas no internet access) to my main vlan (25 with internet). Some VM's on Vlan 20 get better transfer speeds within vlan 20.

I switched a bunch of things to 10G & it runs better but I noticed that with a ton of IP cameras, two NVRS (dont ask), that there was so much traffic on my 1gb switches that my 1gb Verizon FIOS connection was only getting 400 speeds. When I turned off my two NVR's the speeds went to the mid 900 range which is closer to what I would expect. Once the NVR's went back on speeds slowed the whole network down.

When I moved from Cisco 1gb to these 6450/7250 units & made sure the only inter switch connectivity was through the 10gb connections it has been great.

I do have a lag setup so I am at 30gb between switches except:

Verizon Fios ONT to Firewall (1gb) to Unifi switch (1gb fiber) to icx6450.

just some thoughts - not sure if this is helpful.

richtj99 · Jul 3, 2021

Hi - I purchased a 7250-48p - first one came with no working serial port, seller sent a 'tested' replacement - serial does work, as soon as it boots up I get this error:

Code:

U1-MSG: PoE Severe Error: Power being injected on port 1/1/33. No new PDs can get powered on this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

I tried disabling POE on all ports, just port 33, etc. I am not sure what to do or if this is fixable.

I took a photo of the inside (attached).

Code:

ICX7250-48P Router# show inline power

Power Capacity:         Total is 740000 mWatts. Current Free is 740000 mWatts.

Power Allocations:      Requests Honored 48 times


 Port   Admin   Oper    ---Power(mWatts)---  PD Type  PD Class  Pri  Fault/
        State   State   Consumed  Allocated                          Error
--------------------------------------------------------------------------
  1/1/1 On      Off            0          0  n/a      n/a         3  n/a
  1/1/2 On      Off            0          0  n/a      n/a         3  n/a
  1/1/3 On      Off            0          0  n/a      n/a         3  n/a
  1/1/4 On      Off            0          0  n/a      n/a         3  n/a
  1/1/5 On      Off            0          0  n/a      n/a         3  n/a
  1/1/6 On      Off            0          0  n/a      n/a         3  n/a
  1/1/7 On      Off            0          0  n/a      n/a         3  n/a
  1/1/8 On      Off            0          0  n/a      n/a         3  n/a
  1/1/9 On      Off            0          0  n/a      n/a         3  n/a
 1/1/10 On      Off            0          0  n/a      n/a         3  n/a
 1/1/11 On      Off            0          0  n/a      n/a         3  n/a
 1/1/12 On      Off            0          0  n/a      n/a         3  n/a
 1/1/13 On      Off            0          0  n/a      n/a         3  n/a
 1/1/14 On      Off            0          0  n/a      n/a         3  n/a
 1/1/15 On      Off            0          0  n/a      n/a         3  n/a
 1/1/16 On      Off            0          0  n/a      n/a         3  n/a
 1/1/17 On      Off            0          0  n/a      n/a         3  n/a
 1/1/18 On      Off            0          0  n/a      n/a         3  n/a
 1/1/19 On      Off            0          0  n/a      n/a         3  n/a
 1/1/20 On      Off            0          0  n/a      n/a         3  n/a
 1/1/21 On      Off            0          0  n/a      n/a         3  n/a
 1/1/22 On      Off            0          0  n/a      n/a         3  n/a
 1/1/23 On      Off            0          0  n/a      n/a         3  n/a
 1/1/24 On      Off            0          0  n/a      n/a         3  n/a
 1/1/25 On      Off            0          0  n/a      n/a         3  n/a
 1/1/26 On      Off            0          0  n/a      n/a         3  n/a
 1/1/27 On      Off            0          0  n/a      n/a         3  n/a
 1/1/28 On      Off            0          0  n/a      n/a         3  n/a
 1/1/29 On      Off            0          0  n/a      n/a         3  n/a
 1/1/30 On      Off            0          0  n/a      n/a         3  n/a
 1/1/31 On      Off            0          0  n/a      n/a         3  n/a
 1/1/32 On      Off            0          0  n/a      n/a         3  n/a
 1/1/33 On      Off            0          0  n/a      n/a         3  voltage applied from ext src
 1/1/34 On      Off            0          0  n/a      n/a         3  n/a
 1/1/35 On      Off            0          0  n/a      n/a         3  n/a
 1/1/36 On      Off            0          0  n/a      n/a         3  n/a
 1/1/37 On      Off            0          0  n/a      n/a         3  n/a
 1/1/38 On      Off            0          0  n/a      n/a         3  n/a
 1/1/39 On      Off            0          0  n/a      n/a         3  n/a
 1/1/40 On      Off            0          0  n/a      n/a         3  n/a
 1/1/41 On      Off            0          0  n/a      n/a         3  n/a
 1/1/42 On      Off            0          0  n/a      n/a         3  n/a
 1/1/43 On      Off            0          0  n/a      n/a         3  n/a
 1/1/44 On      Off            0          0  n/a      n/a         3  n/a
 1/1/45 On      Off            0          0  n/a      n/a         3  n/a
 1/1/46 On      Off            0          0  n/a      n/a         3  n/a
 1/1/47 On      Off            0          0  n/a      n/a         3  n/a
 1/1/48 On      Off            0          0  n/a      n/a         3  n/a
--------------------------------------------------------------------------
 Total                         0          0

I disable port 33 poe then plug two poe devices into other random ports with POE enabled.

Code:

ICX7250-48P Router# U1-MSG: PoE Severe Error: PD on port 1/1/10 cannot be powered due to power being injected on another port of this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

U1-MSG: PoE Severe Error: PD on port 1/1/3 cannot be powered due to power being injected on another port of this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

Sort of at a loss on what to try?

Any suggestions would be great.

NablaSquaredG · Jul 3, 2021

https://forums.ruckuswireless.com/conversations/ruckus-indoor-aps/poe-error-after-power-cycling-icx-7150-c12p/5fe1ffee28ada6796a64ef24

Probably a hardware failure...

joshdave · Jul 3, 2021

I recently purchased a ICX6450-24p. I have NTP set up, and it properly sets the clock not long after the unit is booted. However I noticed that during boot and shortly after boot, the clock is set to 1969. Not a biggie, but just curious if this is normal for this switch? I tested the battery (BR2032), and it looks good (2.83v).

epicurean · Jul 4, 2021

richtj99 said:
Im wondering if this has to do with the pfsense firewall & the speed of Vlan 1 talking to vlan 80? I notice with my firewall, even with port rules enabled for inter-vlan traffic, there is a speed difference in transferring files from vlan 20 (nas no internet access) to my main vlan (25 with internet). Some VM's on Vlan 20 get better transfer speeds within vlan 20.

I switched a bunch of things to 10G & it runs better but I noticed that with a ton of IP cameras, two NVRS (dont ask), that there was so much traffic on my 1gb switches that my 1gb Verizon FIOS connection was only getting 400 speeds. When I turned off my two NVR's the speeds went to the mid 900 range which is closer to what I would expect. Once the NVR's went back on speeds slowed the whole network down.

When I moved from Cisco 1gb to these 6450/7250 units & made sure the only inter switch connectivity was through the 10gb connections it has been great.

I do have a lag setup so I am at 30gb between switches except:

Verizon Fios ONT to Firewall (1gb) to Unifi switch (1gb fiber) to icx6450.

just some thoughts - not sure if this is helpful.

I do have another 6450 (non POE) and a unifi switch 16 XB Is it possible to use them to offset this loss in bandwidth from the POE cameras
? or is there some setting in pfsense that can mitigate this?

fohdeesha · Jul 4, 2021

richtj99 said:

Hi - I purchased a 7250-48p - first one came with no working serial port, seller sent a 'tested' replacement - serial does work, as soon as it boots up I get this error:

Code:

U1-MSG: PoE Severe Error: Power being injected on port 1/1/33. No new PDs can get powered on this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

I tried disabling POE on all ports, just port 33, etc. I am not sure what to do or if this is fixable.

I took a photo of the inside (attached).

Code:

ICX7250-48P Router# show inline power

Power Capacity:         Total is 740000 mWatts. Current Free is 740000 mWatts.

Power Allocations:      Requests Honored 48 times


Port   Admin   Oper    ---Power(mWatts)---  PD Type  PD Class  Pri  Fault/
        State   State   Consumed  Allocated                          Error
--------------------------------------------------------------------------
  1/1/1 On      Off            0          0  n/a      n/a         3  n/a
  1/1/2 On      Off            0          0  n/a      n/a         3  n/a
  1/1/3 On      Off            0          0  n/a      n/a         3  n/a
  1/1/4 On      Off            0          0  n/a      n/a         3  n/a
  1/1/5 On      Off            0          0  n/a      n/a         3  n/a
  1/1/6 On      Off            0          0  n/a      n/a         3  n/a
  1/1/7 On      Off            0          0  n/a      n/a         3  n/a
  1/1/8 On      Off            0          0  n/a      n/a         3  n/a
  1/1/9 On      Off            0          0  n/a      n/a         3  n/a
1/1/10 On      Off            0          0  n/a      n/a         3  n/a
1/1/11 On      Off            0          0  n/a      n/a         3  n/a
1/1/12 On      Off            0          0  n/a      n/a         3  n/a
1/1/13 On      Off            0          0  n/a      n/a         3  n/a
1/1/14 On      Off            0          0  n/a      n/a         3  n/a
1/1/15 On      Off            0          0  n/a      n/a         3  n/a
1/1/16 On      Off            0          0  n/a      n/a         3  n/a
1/1/17 On      Off            0          0  n/a      n/a         3  n/a
1/1/18 On      Off            0          0  n/a      n/a         3  n/a
1/1/19 On      Off            0          0  n/a      n/a         3  n/a
1/1/20 On      Off            0          0  n/a      n/a         3  n/a
1/1/21 On      Off            0          0  n/a      n/a         3  n/a
1/1/22 On      Off            0          0  n/a      n/a         3  n/a
1/1/23 On      Off            0          0  n/a      n/a         3  n/a
1/1/24 On      Off            0          0  n/a      n/a         3  n/a
1/1/25 On      Off            0          0  n/a      n/a         3  n/a
1/1/26 On      Off            0          0  n/a      n/a         3  n/a
1/1/27 On      Off            0          0  n/a      n/a         3  n/a
1/1/28 On      Off            0          0  n/a      n/a         3  n/a
1/1/29 On      Off            0          0  n/a      n/a         3  n/a
1/1/30 On      Off            0          0  n/a      n/a         3  n/a
1/1/31 On      Off            0          0  n/a      n/a         3  n/a
1/1/32 On      Off            0          0  n/a      n/a         3  n/a
1/1/33 On      Off            0          0  n/a      n/a         3  voltage applied from ext src
1/1/34 On      Off            0          0  n/a      n/a         3  n/a
1/1/35 On      Off            0          0  n/a      n/a         3  n/a
1/1/36 On      Off            0          0  n/a      n/a         3  n/a
1/1/37 On      Off            0          0  n/a      n/a         3  n/a
1/1/38 On      Off            0          0  n/a      n/a         3  n/a
1/1/39 On      Off            0          0  n/a      n/a         3  n/a
1/1/40 On      Off            0          0  n/a      n/a         3  n/a
1/1/41 On      Off            0          0  n/a      n/a         3  n/a
1/1/42 On      Off            0          0  n/a      n/a         3  n/a
1/1/43 On      Off            0          0  n/a      n/a         3  n/a
1/1/44 On      Off            0          0  n/a      n/a         3  n/a
1/1/45 On      Off            0          0  n/a      n/a         3  n/a
1/1/46 On      Off            0          0  n/a      n/a         3  n/a
1/1/47 On      Off            0          0  n/a      n/a         3  n/a
1/1/48 On      Off            0          0  n/a      n/a         3  n/a
--------------------------------------------------------------------------
Total                         0          0

I disable port 33 poe then plug two poe devices into other random ports with POE enabled.

Code:

ICX7250-48P Router# U1-MSG: PoE Severe Error: PD on port 1/1/10 cannot be powered due to power being injected on another port of this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

U1-MSG: PoE Severe Error: PD on port 1/1/3 cannot be powered due to power being injected on another port of this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

Sort of at a loss on what to try?

Any suggestions would be great.

yeah that poe board is fried, tell the seller he's a dingus

fohdeesha · Jul 4, 2021

joshdave said:
I recently purchased a ICX6450-24p. I have NTP set up, and it properly sets the clock not long after the unit is booted. However I noticed that during boot and shortly after boot, the clock is set to 1969. Not a biggie, but just curious if this is normal for this switch? I tested the battery (BR2032), and it looks good (2.83v).

normal

fohdeesha · Jul 4, 2021

big improvements & features coming to the guide soon

richtj99 · Jul 4, 2021

epicurean said:
I do have another 6450 (non POE) and a unifi switch 16 XB Is it possible to use them to offset this loss in bandwidth from the POE cameras
? or is there some setting in pfsense that can mitigate this?

What are the specs of your PFsense box? How is the CPU / Mem usage?

Unifi XG (10gb?) - What VMS/NVR for the cameas? If it is Blue Iris - what are the stream megabits?

Is it 1 vlan for you 1 vlan for cameras? I am happy to help where I can but switching to 10gb was pretty immediate for me. Again my vlans are all set with all cameras on the same camera vlan.

fohdeesha said:
yeah that poe board is fried, tell the seller he's a dingus

Yeah I figured - bummer - can I safely use it as a non-poe unit? I can make an offer on it vs making him pay for shipping again. Then again I am not sure if a 7250-48p with POE disabled will use more electric than its worth. I want to make sure the rest of the switch isnt going to blow up. can I take the POE part of the board out or swap it?

epicurean · Jul 4, 2021

richtj99 said:
What are the specs of your PFsense box? How is the CPU / Mem usage?

Unifi XG (10gb?) - What VMS/NVR for the cameas? If it is Blue Iris - what are the stream megabits?

Is it 1 vlan for you 1 vlan for cameras? I am happy to help where I can but switching to 10gb was pretty immediate for me. Again my vlans are all set with all cameras on the same camera vlan.

My pfsense is an i3 4xxxT CPU with 16GB, on an supermicro X10SLV motherboard
Yes, the Unifi 16XG has SFP+ ports and 4 10 GB RJ45 ports. I do have blue iris, but I yet to connec yet and trying to figure out the network bog down. The blue iris PC is a dedicated 1151 motherboard on a 1GB connection.

My cameras are on vlan 80. I have a different vlan for IOT devices. All other devices are on my main LAN

SuperMiguel · Jul 5, 2021

gregsachs said:
For a 3 ft run I'd use a dac, more robust. Stay at 5m or less, the 6450 doesn't like 10m dacs

SFP-H10GB-CU1.5M Cisco Compatible 10G DAC Cable | eBay

For sale is aSFP-H10GB-CU1.5M Cisco Compatible. Local pick up is available and always free.

www.ebay.com

Any recommendation on network cards?

Was thinking to get these: 0N20KJ DELL BROADCOM 57810 10GB DUAL PORT PCI-E SFP+ NETWORK CARD N20KJ Y40PH | eBay

Propaganda · Jul 5, 2021

What for a icx7250, what is the logic level of miniusb/uart? rs232/TTL(1.8V)/TTL(3.3V)/TTL(5V)/other? Thanks!

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Active Member

New Member

Active Member

Active Member

New Member

Active Member

Active Member

New Member

Member

Member

Attachments

Well-Known Member

New Member

Active Member

Kaini Industries

Kaini Industries

Kaini Industries

Member

Active Member

New Member

Member