Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Got all my SFPs and fiber this afternoon, so I get to pull the run from rack to 7150 in my office, Saturday will be the big day of tearing down the old LAGs, removing a third of the copper from my rack, and swapping the I350-T4 in the server to a Mellanox ConnectX-3 EN. A client has a pair of R720Xd servers soon to be removed, so I'll be grabbing those and installing 10 or 25 Gb capable rNDCs (if available).

Does a 40GQ module on a 7450 support breakout? I can't find a good answer, but the brochure for the 7450 line lists breakout capable QSFPs as supported...
Code:
40G-QSFP-ESR4 40GBASE-ESR4 QSFP+ optic (MTP 1×8 or 1×12), 300 m over MMF, (10GBASE-SR compatible, breakout).
40G-QSFP-SR4-INT 40GBASE-SR4 QSFP+ optic (MTP 1×8 or 1×12), 100 m over MMF (10GBASE-SR compatible, breakout-capable).
 

Vesalius

Active Member
Nov 25, 2019
252
190
43
Got my lab up and running, so fare it’s working great!!!

p2p link between an ER-4 and the switch so most routing is done on the switch

icx7150-48zp with two R310, it’s going to replace my edgeswitch and ubiquitous access points



View attachment 16289
Get anywhere on a fan mod for these. A little too noisy for me. Did yours come with rack mounts? Mine did not.
 

nerdalertdk

Fleet Admiral
Mar 9, 2017
228
118
43
::1
Get anywhere on a fan mod for these. A little too noisy for me. Did yours come with rack mounts? Mine did not.
no not yet, but it’s in a rack in my basement so not really that big of a problem would also say my hp dl20 are louder

No rack mount, but I had some from an edgeswitch they fit nicely on
 

Vesalius

Active Member
Nov 25, 2019
252
190
43
no not yet, but it’s in a rack in my basement so not really that big of a problem would also say my hp dl20 are louder

No rack mount, but I had some from an edgeswitch they fit nicely on
If you move to FastIron 08.0.95 you might consider trying "inline power poe-ha" on the ports powering those Ruckus AP. That should keep them powered during a reload of your 7150. That plus a UPS should keep them happy. They can take a while to rebuild the unleashed network any time the power is reset.

 

Vesalius

Active Member
Nov 25, 2019
252
190
43
I've been holding off on updating the guide to use 8030u as initially the brocade subreddit and the ICX usergroup on Ruckus's support site kept giving vague reports that it was "buggy" and to just stay on 8030t. However I've been running it at home and in production for about a month now and haven't ran into any issues so I will probably upgrade the guide to it soon.

As for the icx7xxx series guides, still deciding what to do there as I'd like to recommend the latest 8092 train, however it's distributed differently and HAS to be flashed using a UFI image. The fun part: versions before 8080e don't support flashing UFI images. This makes the guide very difficult to write as the process will completely depend on what version your switch came with, I'd basically have to write two guides: one for people on firmware earlier than 8080e, in which case they'll have to jump to 8080e first, then to 8092, and another guide for people already on 8080e or later

I take pride in how simple and straightforward my docu is and this kind of mess/multiple paths breaks pretty much all my rules, so I'm trying to work out something simpler that covers everyone's switch regardless of state. It will probably involve flashing one of the later/latest bootloaders, then flashing the latest UFI 8092 image from within the bootloader - that way it won't rely on the user having a specific version of fastiron with UFI flashing support.
I am fairly certain you have seen this or already know it, but just in case these are the options Ruckus recommends moving to the new UFI images that no longer have separate Bootrom installs. Seems like TFTP option 1 might work for your documentation and you would need to add whatever UFI version (8092?) you see as best in addition to the 08.0.80e bin and bootrom you already have zipped up. But obviously more words would have to used to help the novice understand what to do.

 
Last edited:

nerdalertdk

Fleet Admiral
Mar 9, 2017
228
118
43
::1
If you move to FastIron 08.0.95 you might consider trying "inline power poe-ha" on the ports powering those Ruckus AP. That should keep them powered during a reload of your 7150. That plus a UPS should keep them happy. They can take a while to rebuild the unleashed network any time the power is reset.

I actually did run 08.0.95 at first, but downgraded it to 08.0.90d since that's the recommended firmware from ruckus
 

ip64.uk

New Member
Mar 22, 2020
2
5
3
Thank you 'dodgy route' for the post that inspired me to fan-mod my icx6610 48x poe, and thank you foheesha for helping me get my switch licensed.

It took a couple of weeks to get all the parts in, but once they all arrived I got straight to work modding the switch.

The key to getting all this to work is the 'NE555 Pulse Square Wave Signal Generator' - this fakes the tach signal so the switch thinks the original fans are connected. I picked up the one below from ebay for £2.30. You can adjust the frequency and duty cycle using the two pots (looking at the picture below, left is frequency, right is the duty cycle). To calculate the frequency to use, use this calculation: RPM = (FREQ x 60) / 2 .. I used 670Hz which is 20100rpm at a 50% duty cycle. You should be able to mod any switch using a pulse generator like this one. 12v, gnd, and tach pinout can be found easily by following the wires from one of the fans.

s-l1600.jpg

I used two Bitfenix Spectre Pro 230mm fans instead of the Noctua NF-A20 as they were half the price. According to Bitfenix, these fans also have greater CFM and static pressure. I run the fans at 12v full speed - as due to the rpm and size, they are pretty much silent anyway.

A few other changes I made (compared to the post by 'dodgy route'), I removed a metal divider on the side of the switch without a psu - this was done to improve airflow. I have also hotglued a strip of acrylic across the second psu and fan bays - testing to see if it improves cooling by forcing air out over the 'busy' side of the switch (rather than straight out the back). Instead of putting wires directly into the fan-bay connector on the mainboard, I decided to mod one of the fan trays instead (my switch shipped with 2 psu and 2 fan trays). When it comes to modding the psu, I decided to cut the entire top out of the (plastic/paper?) 'shield/cover' - this was also done to improve airflow.

All credit goes to 'dodgy route'!!! If you are interested in modding your switch, I highly recommend reading their post and checking out their google photos album (they took photos of the whole process). Click here to view their post

6610-fan-mod.jpg

After a couple of hours idling, the temperatures look good! (ambient room temp 25*c).

Code:
ICX6610-48P-Router>show chassis
The stack unit 1 chassis info:

Power supply 1 not present
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  T62
        Firmware Ver:    B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 not present
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 38.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->84       deg-C
                Speed 2:       79<-----> 87 deg-C (shutdown)

Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 31.0 deg-C
MAC 2 Temperature Readings:
        Current temperature : 38.0 deg-C
CPU Temperature Readings:
        Current temperature : 33.0 deg-C
sensor A Temperature Readings:
        Current temperature : 21.5 deg-C
sensor B Temperature Readings:
        Current temperature : 28.5 deg-C
sensor C Temperature Readings:
        Current temperature : 28.5 deg-C
stacking card Temperature Readings:
        Current temperature : 37.0 deg-C
        Warning level.......: 77.0 deg-C
        Shutdown level......: 87.0 deg-C
 

mshook

New Member
Jun 9, 2020
14
10
3
Thank you 'dodgy route' for the post that inspired me to fan-mod my icx6610 48x poe, and thank you foheesha for helping me get my switch licensed.
...
That is just brilliant. I should do that with my ICX6450, the only issue is the cover also covers the sides, so I'd have to do acrylic for side panels as well.
 
  • Like
Reactions: ip64.uk

ip64.uk

New Member
Mar 22, 2020
2
5
3
How would you rate the noise before vs after?
In my opinion, before it was way too loud for 24/7 (if in the same room).. Now it's no louder than my desktop computer.. but it really depends on how sensitive you are to noise - but without a doubt, it's a massive improvement over the stock fans (in both noise output and cooling performance). The good thing about large fans, is that they move alot of air at lower rpms - if you do hear them, its usually a 'low' frequency sound rather than an annoying high pitched one. Must also remember, I have the 48 port version with POE, so this switch might be louder than other models
 

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
Had my first real problem with my ICX6610 last night.

After setting up 2 ipv6 access-lists with about 40 ACLs each and assigning them to their respective ves, the switch rebooted itself after about 3 minutes. When it came back online, it did it again (same config). Then it did it again. No errors in the console, just a sudden reset.

I was able to login and remove the ACLs quick enough on the 4th reboot and it's been fine since.

Has anyone else noticed any reliability issues with IPv6 stuff? It seems to be that the 8.0.30 train is fairly immature with regards to IPv6.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Had my first real problem with my ICX6610 last night.

After setting up 2 ipv6 access-lists with about 40 ACLs each and assigning them to their respective ves, the switch rebooted itself after about 3 minutes. When it came back online, it did it again (same config). Then it did it again. No errors in the console, just a sudden reset.

I was able to login and remove the ACLs quick enough on the 4th reboot and it's been fine since.

Has anyone else noticed any reliability issues with IPv6 stuff? It seems to be that the 8.0.30 train is fairly immature with regards to IPv6.
well that's wild. Never seen that before, I have a bunch of v6 ACLs on a stack of 2 6610's here at home and never ran into any issues. Could you post the problematic config, and what firmware version are you on
 

HaxSmash

New Member
Oct 15, 2020
8
2
3
I'm hoping for some help. I'm a networking newbie, and after fighting with

it a few hours, I'm just not sure what to do. If anything I get the feeling that what I am trying to do is probably just dumb to begin with, so there is that.

I am hoping to be able to utilize layer 3 switching on my 6450, and use my existing Ubnt USG as my internet firewall / router.

What I have setup currently for testing is as follows.

On the USG, I have 2 new networks defined on the second LAN interface.
- VLAN 99 192.168.99.0/30 with the USG on 192.168.99.2. This is acting as my fallback network for the times that I inevitably break my SSH connection to help prevent me from having to venture to the basement with a console cable.
- VLAN 25, 192.168.25.0/24 USG listening on 192.168.25.2, DHCP server enabled, default gateway setup for 192.168.25.1

On the 6450:

I have the link between the USG and the 6450 on 1/1/1. This is tagged in VLAN 25 and VLAN99.
I have port 1/1/2 untagged in vlan 25, this is where i'm connecting my test machine. I have a helper-address set to the USG gateway for DHCP.
I have the default gateway set to 192.168.25.2

Code:
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 25 by port
 tagged ethe 1/1/1
 untagged ethe 1/1/2
 router-interface ve 25
 
vlan 99 by port
 tagged ethe 1/1/1
 router-interface ve 99

!
interface ve 25
 ip address 192.168.25.1 255.255.255.0
 ip helper-address 1 192.168.25.2
!

interface ve 99
 ip address 192.168.99.2 255.255.255.0
!

ip dns server-address 192.168.25.2
ip route 0.0.0.0/0 192.168.25.2
The issue I am having is, when configured like this, only one of my VLAN's on the 6450 is able to route outside the switch at any given point in time. Configured as shown above, my test device is able to access the internet, and the rest of my network (on the other USG lan port). DHCP works as expected, all is wonderful.

However, my VLAN99 can't access anything. For example, if I ping google.com source 192.168.99.2, I just get request timeouts. It is the same for anything I try to do when sourcing from the 99.2 address. If I switch the default gateway of the switch to be 192.168.99.1, my VLAN 99 is able to communicate to the outside world, but VLAN25 becomes effectively dead.

I'm just not sure how to configure things to have more than one VLAN functioning on my switch while using layer 3 switching. If I just use my USG as the default gateway for my laptop, everything works perfectly fine. My question is... am I trying to do something stupid? My reasoning for trying to do this is that I want to be able to do 10G switching across VLANs within my network and not be bottle-necked by the USG's 1gbe connection. I also want to continue using as much as the unifi stuff as I can, since it makes it easy to deal with IP reservations, and monitor traffic.
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
I have a helper-address set to the USG gateway for DHCP.
There's no need for an helper address if the DHCP server is on the same broadcast domain (ie. the same VLAN).

if I ping google.com source 192.168.99.2, I just get request timeouts. It is the same for anything I try to do when sourcing from the 99.2 address.
That might be normal because the network 192.168.99.0/24 is locally connected on the USG, so it would forward it directly and not via 192.168.25.1, so if you've defined some policies that accept traffic from that address, those won't be matched.

Try with a traceroute, you'll see all the hops taken by a packet.

It's really a bad idea to mix up routing and switching; a routed network must not be defined on any upstream router interface, only in its route table.
 

HaxSmash

New Member
Oct 15, 2020
8
2
3
It's really a bad idea to mix up routing and switching; a routed network must not be defined on any upstream router interface, only in its route table.
Ok so that goes along with my gut feeling that I'm trying to do something that I shouldn't, my ignorance showing.

So If I'm following my options are:

a) Continue to use the USG to define all my vlans, and just use the brocade in more of a dumb layer 2 mode?

b) Scrap the vlans and just use one big flat dumb network (aka give in to my lack of knowledge)

b) Make the USG less aware of the downstream switching, define everything on the brocade except for a simple uplink / default route? I guess i'm missing something here around where the NAT happens and the routing. So much to learn.

I started writing this reply thinking I knew a path forward just to realize I'm further away than I thought. I'll have to search some more for some examples.
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
my gut feeling that I'm trying to do something that I shouldn't, my ignorance showing.
Don't worry too much, everyone has learned the lessons the hard way by breaking something... at worst case, simply don't save to startup config and pull the plug :p

I guess i'm missing something here around where the NAT happens and the routing. So much to learn.
Now I am the one missing someting as I don't know about how the USG handles NAT for networks not defined... I don't think it goes too well.

A trick for learn without many trips to the basement is, provided you have a PC capable of vlan tagging, define the "transit network" (192.168.25.0/24) as a tagged VLAN on the interface "to the desk" (or even better, to another interface if you have two wires), define the "LAN" 192.168.99.0/24 inside the USG but do not assign it to the interface, and add a route for 192.168.99.0/24 via 192.168.25.1... everything """should""" work. If this is the case, grow as you like; if not, switch on the tagged VLAN and you can reach the interwebs and the configuration consoles.
Oh, and remember to define the DHCP helper on ve 99, otherwise you won't obtain a lease.

I started writing this reply thinking I knew a path forward just to realize I'm further away than I thought. I'll have to search some more for some examples.
Don't feel too bad, the hard trick is to learn it the first time, every other network is really a "rinse and repeat" situation.
The controller thingy of the UniFI world, in my opinion, does more harm than good in those situations, as it hides from sight the plain basic things, in a "let's simplify for the masses" way that I feel completely dumb, but oh-well...

Don't give up, you'll have to break stuff to learn how it works, don't be ashamed of it.
 

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
well that's wild. Never seen that before, I have a bunch of v6 ACLs on a stack of 2 6610's here at home and never ran into any issues. Could you post the problematic config, and what firmware version are you on
I'm running version 08.0.30uT7f3.

Here's the 2 ACLs that caused me issues.
Code:
ipv6 access-list iot-v6
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 2605:a000:d401:7a03::1 eq ssh log
deny tcp any host 2605:a000:d401:7a03::1 eq telnet log
deny tcp any host 2605:a000:d401:7a03::1 eq http log
deny tcp any host 2605:a000:d401:7a03::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:a000:d401:7a03::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:a000:d401:7a26::3 eq dns
permit tcp any host 2605:a000:d401:7a26::3 eq dns
permit udp any host 2605:a000:d401:7a26::5 eq dns
permit tcp any host 2605:a000:d401:7a26::5 eq dns                            
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 2605:a000:d401:7a26::81
permit udp any eq snmp-trap host 2605:a000:d401:7a26::81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ipv6 any 2605:a000:d401:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
!
ipv6 access-list voip-v6
remark DENY ADMIN ACCESS TO SWITCH
deny tcp any host 2605:a000:d401:7a02::1 eq ssh log
deny tcp any host 2605:a000:d401:7a02::1 eq telnet log
deny tcp any host 2605:a000:d401:7a02::1 eq http log
deny tcp any host 2605:a000:d401:7a02::1 eq ssl log
remark ALLOW SAME VLAN TRAFFIC
permit ipv6 any 2605:a000:d401:7a02::/64
remark ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
remark ALLOW ICMP
permit icmp any any
remark ALLOW ESTABLISHED TCP TRAFFIC                            
permit tcp any any established
remark ALLOW DNS REQUESTS
permit udp any host 2605:a000:d401:7a26::3 eq dns
permit tcp any host 2605:a000:d401:7a26::3 eq dns
permit udp any host 2605:a000:d401:7a26::5 eq dns
permit tcp any host 2605:a000:d401:7a26::5 eq dns
remark ALLOW RETURN OF SNMP TRAFFIC TO LIBRENMS SERVER
permit udp any eq snmp host 2605:a000:d401:7a26::81
permit udp any eq snmp-trap host 2605:a000:d401:7a26::81
remark DENY ALL OTHER INTER-VLAN TRAFFIC
deny ipv6 any 2605:a000:d401:7a00::/56 log
remark ALLOW REMAINING TRAFFIC
permit ipv6 any any
EDIT:

I gave it another shot tonight, and so far no problems. I'm testing IPv6 ACLs on these two networks because there's only a handful of devices on them that support IPv6 before I roll out more ACLs to my other VLANs.
 
Last edited:

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
Ok so that goes along with my gut feeling that I'm trying to do something that I shouldn't, my ignorance showing.

So If I'm following my options are:

a) Continue to use the USG to define all my vlans, and just use the brocade in more of a dumb layer 2 mode?

b) Scrap the vlans and just use one big flat dumb network (aka give in to my lack of knowledge)

b) Make the USG less aware of the downstream switching, define everything on the brocade except for a simple uplink / default route? I guess i'm missing something here around where the NAT happens and the routing. So much to learn.

I started writing this reply thinking I knew a path forward just to realize I'm further away than I thought. I'll have to search some more for some examples.

Option B (the first one) is closer to what you want, probably. Might as well not tag VLAN traffic between the switch and USG if it might make things easier. I don't tag traffic between my switch and OPNSense just for simplicity's sake.

Try this set up....

USG
  • LAN IP: 192.168.123.1/30 (no VLANs)
  • *Create static route: 192.168.0.0/16 via 192.168.123.2
  • *NAT the whole 192.168.0.0/16 network
* I have no idea how to do this on the USG

Switch
Code:
vlan 10 name Trusted
tagged ethernet 1/1/2 to 1/1/24
router-interface ve 10

vlan 20 name Servers
tagged ethernet 1/1/2 to 1/1/24
router-interface ve 20

vlan 30 name Guest
tagged ethernet 1/1/2 to 1/1/24
router-interface ve 30


ip route 0.0.0.0/0 192.168.123.1

interface ethernet 1/1/1
port-name USG-uplink
route-only
ip address 192.168.123.2/30

interface loopback 1
port-name Management
ip address 192.168.0.1/32

interface ve 10
port-name Trusted
ip address 192.168.10.1/24
ip helper-address 192.168.123.1

interface ve 20
port-name Servers
ip address 192.168.20.1/24
ip helper-address 192.168.123.1

interface ve 30
port-name Guest
ip address 192.168.30.1/24
ip helper-address 192.168.123.1

interface ethernet 1/1/2 to 1/1/24
dual-mode 30
This is a basic setup, but something to experiment with.

It's a smart idea to have a management IP on a loopback interface. Is 1/1/1 is down/disabled, than 192.168.123.1 is inaccessible from any other configured switch addresses, but 192.168.0.1 will always be up.

If the USG will not hand out addresses on the non-local networks (anything outside of 192.168.123.0/30, than you must set up a proper DHCP server and adjust the helper-address appropriately.
 
Last edited:

HaxSmash

New Member
Oct 15, 2020
8
2
3
Option B (the first one) is closer to what you want, probably. Might as well not tag VLAN traffic between the switch and USG if it might make things easier. I don't tag traffic between my switch and OPNSense just for simplicity's sake.
...
I did play around with this some, and ended up finding a setup that appears to do most of what I would like. For experimentation, I allocated a /20 block to play with. 192.168.48.0/20


As suggested, in the Unifi controller I configured the following:

- The transit/uplink network on LAN2, as 192.168.123.1/30 no DHCP services.
- Created a static route 192.168.48.0/20, next hop 192.168.123.2

On the Brocade:
- I wasn't able to figure out a way to get it to let me assign an IP directly to e 1/1/1, even without it tagged in any network, and no ip assigned to VE1. I ended up just assigning the 192.168.123.2 IP to VE1, which appears to be working.
- I set the default gateway to 192.168.123.1
- I then created my VLANs wholly within the switch, using the DHCP server built into the switch, as nothing I tried would get the USG to give out IP, since it doesn't know about those VLAN networks.

With things setup like this, I can plug into either my "50" vlan or "60" vlan on the switch, and get the correct IP, and can reach the rest of my network, and outside world correctly. The one oddity is that the Unifi controller lists the IP's as being owned by the MAC of the switch, but that isn't really an issue as far as I can tell.

Thank you so much for your suggestions, they did help guide me towards something that worked!
 

victimofareload

New Member
Nov 10, 2020
6
0
1
Hey everyone, Sort of long time lurker. first time poster. I've been using ebay cisco switches for a few years now and have always been happy. But want to expand my knowledge of other platforms as well as bring 10G to the home lab.

I got a ICX6610-48P-E on the way. Is getting licenses still a thing? Got the switch from a friend of a friend of a friend and I'm pretty sure it has no license on it. But I'll know more when I get my hands on it.

Thanks!