Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
2,616
2,810
113
32
fohdeesha.com
VEs cannot grab an address via DHCP, at least on the icx6xxx series. If you plug in an otherwise blankly configged switch to a network with a dhcp server, fastiron will grab a lease and throw it on the physical port you've plugged in. One of the first steps in my config guide in the OP of this thread is to disable dhcp-client so this doesn't happen, and it instructs you to set up a VE with a static IP so you can manage it in-band. Not sure why you'd want a dynamically assigned address for management of something as important as a core switch. Just pick an unused IP outside of your dhcp range to give the switch, and follow the config guide
 

rootpeer

Member
Oct 19, 2019
71
16
8
VEs cannot grab an address via DHCP, at least on the icx6xxx series. If you plug in an otherwise blankly configged switch to a network with a dhcp server, fastiron will grab a lease and throw it on the physical port you've plugged in. One of the first steps in my config guide in the OP of this thread is to disable dhcp-client so this doesn't happen, and it instructs you to set up a VE with a static IP so you can manage it in-band. Not sure why you'd want a dynamically assigned address for management of something as important as a core switch. Just pick an unused IP outside of your dhcp range to give the switch, and follow the config guide
Yeah I actually followed your guides for resetting and setting up the switch, I just thought I would try with DHCP first. I am using pfSense for DHCP and DNS so I set my static DHCP mappings there and then access my stuff through their hostnames. I figured that this is not really the way to use these switches since they are L3, I just thought I would try it my usual way first. So to recap, it seems like it is working as designed and we really need to set a static IP since that was how it was supposed to be used.

Please post a "show run" output, but to me there's something strange on your configuration...

Have you issued a "route-only" on ethe 1/1/1? In case, yes, in can't be tagged or untagged, as it will work as a pure layer3 only interface, no switching operations can be done. But it may cause havoc on the relative vlan, as it shouldn't forward traffic to other clients -- I've never used it, I must be sincere with you.
But you're right, it seems that only route only ports can obtain a dhcp lease, not the virtual interfaces. But, they should be the gateway of the relative vlan/broadcast domain, so it is a reasonable assumption that they must have a static IP only.
I don't think I did, I think it does it (or something else) automatically on DHCP.

Code:
SSH@icx645024p(config-vlan-200)#show run
Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
  module 1 icx6450-24p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 200 name family by port
 tagged ethe 1/1/2 to 1/1/3
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console                                               
hostname icx645024p
ip dns domain-list thes
ip dns server-address 10.0.10.1
!
no telnet server
username root password .....
!
!
!
!
!
interface ethernet 1/1/1
 ip address 10.0.10.202 255.255.255.0 dynamic
!
!
!
!
!
!
!
!
!
end
 
  • Like
Reactions: fohdeesha

infoMatt

Active Member
Apr 16, 2019
222
99
28
I don't think I did, I think it does it (or something else) automatically on DHCP.
Yes, it's a mainly blank config. I have to apologize, I hadn't noticed the behavior that @fohdeesha said, as every time I've reconfigured the switch I've assigned a static IP to a VE. Sorry :(
I was thinking at the route-only interfaces as the only way a port could have a directly assigned IP address, but I had forgotten the dhcp client.
 

nerdalertdk

Fleet Admiral
Mar 9, 2017
228
118
43
::1
Got my lab up and running, so fare it’s working great!!!

p2p link between an ER-4 and the switch so most routing is done on the switch

icx7150-48zp with two R310, it’s going to replace my edgeswitch and ubiquitous access points



40D5C836-318D-4D67-854B-4B8B68F9E3CB.jpeg
 

LodeRunner

Active Member
Apr 27, 2019
489
210
43
Got all my SFPs and fiber this afternoon, so I get to pull the run from rack to 7150 in my office, Saturday will be the big day of tearing down the old LAGs, removing a third of the copper from my rack, and swapping the I350-T4 in the server to a Mellanox ConnectX-3 EN. A client has a pair of R720Xd servers soon to be removed, so I'll be grabbing those and installing 10 or 25 Gb capable rNDCs (if available).

Does a 40GQ module on a 7450 support breakout? I can't find a good answer, but the brochure for the 7450 line lists breakout capable QSFPs as supported...
Code:
40G-QSFP-ESR4 40GBASE-ESR4 QSFP+ optic (MTP 1×8 or 1×12), 300 m over MMF, (10GBASE-SR compatible, breakout).
40G-QSFP-SR4-INT 40GBASE-SR4 QSFP+ optic (MTP 1×8 or 1×12), 100 m over MMF (10GBASE-SR compatible, breakout-capable).
 

Vesalius

Active Member
Nov 25, 2019
224
160
43
Got my lab up and running, so fare it’s working great!!!

p2p link between an ER-4 and the switch so most routing is done on the switch

icx7150-48zp with two R310, it’s going to replace my edgeswitch and ubiquitous access points



View attachment 16289
Get anywhere on a fan mod for these. A little too noisy for me. Did yours come with rack mounts? Mine did not.
 

nerdalertdk

Fleet Admiral
Mar 9, 2017
228
118
43
::1
Get anywhere on a fan mod for these. A little too noisy for me. Did yours come with rack mounts? Mine did not.
no not yet, but it’s in a rack in my basement so not really that big of a problem would also say my hp dl20 are louder

No rack mount, but I had some from an edgeswitch they fit nicely on
 

Vesalius

Active Member
Nov 25, 2019
224
160
43
no not yet, but it’s in a rack in my basement so not really that big of a problem would also say my hp dl20 are louder

No rack mount, but I had some from an edgeswitch they fit nicely on
If you move to FastIron 08.0.95 you might consider trying "inline power poe-ha" on the ports powering those Ruckus AP. That should keep them powered during a reload of your 7150. That plus a UPS should keep them happy. They can take a while to rebuild the unleashed network any time the power is reset.

 

Vesalius

Active Member
Nov 25, 2019
224
160
43
I've been holding off on updating the guide to use 8030u as initially the brocade subreddit and the ICX usergroup on Ruckus's support site kept giving vague reports that it was "buggy" and to just stay on 8030t. However I've been running it at home and in production for about a month now and haven't ran into any issues so I will probably upgrade the guide to it soon.

As for the icx7xxx series guides, still deciding what to do there as I'd like to recommend the latest 8092 train, however it's distributed differently and HAS to be flashed using a UFI image. The fun part: versions before 8080e don't support flashing UFI images. This makes the guide very difficult to write as the process will completely depend on what version your switch came with, I'd basically have to write two guides: one for people on firmware earlier than 8080e, in which case they'll have to jump to 8080e first, then to 8092, and another guide for people already on 8080e or later

I take pride in how simple and straightforward my docu is and this kind of mess/multiple paths breaks pretty much all my rules, so I'm trying to work out something simpler that covers everyone's switch regardless of state. It will probably involve flashing one of the later/latest bootloaders, then flashing the latest UFI 8092 image from within the bootloader - that way it won't rely on the user having a specific version of fastiron with UFI flashing support.
I am fairly certain you have seen this or already know it, but just in case these are the options Ruckus recommends moving to the new UFI images that no longer have separate Bootrom installs. Seems like TFTP option 1 might work for your documentation and you would need to add whatever UFI version (8092?) you see as best in addition to the 08.0.80e bin and bootrom you already have zipped up. But obviously more words would have to used to help the novice understand what to do.

 
Last edited:

nerdalertdk

Fleet Admiral
Mar 9, 2017
228
118
43
::1
If you move to FastIron 08.0.95 you might consider trying "inline power poe-ha" on the ports powering those Ruckus AP. That should keep them powered during a reload of your 7150. That plus a UPS should keep them happy. They can take a while to rebuild the unleashed network any time the power is reset.

I actually did run 08.0.95 at first, but downgraded it to 08.0.90d since that's the recommended firmware from ruckus
 

ip64.uk

New Member
Mar 22, 2020
2
5
3
Thank you 'dodgy route' for the post that inspired me to fan-mod my icx6610 48x poe, and thank you foheesha for helping me get my switch licensed.

It took a couple of weeks to get all the parts in, but once they all arrived I got straight to work modding the switch.

The key to getting all this to work is the 'NE555 Pulse Square Wave Signal Generator' - this fakes the tach signal so the switch thinks the original fans are connected. I picked up the one below from ebay for £2.30. You can adjust the frequency and duty cycle using the two pots (looking at the picture below, left is frequency, right is the duty cycle). To calculate the frequency to use, use this calculation: RPM = (FREQ x 60) / 2 .. I used 670Hz which is 20100rpm at a 50% duty cycle. You should be able to mod any switch using a pulse generator like this one. 12v, gnd, and tach pinout can be found easily by following the wires from one of the fans.

s-l1600.jpg

I used two Bitfenix Spectre Pro 230mm fans instead of the Noctua NF-A20 as they were half the price. According to Bitfenix, these fans also have greater CFM and static pressure. I run the fans at 12v full speed - as due to the rpm and size, they are pretty much silent anyway.

A few other changes I made (compared to the post by 'dodgy route'), I removed a metal divider on the side of the switch without a psu - this was done to improve airflow. I have also hotglued a strip of acrylic across the second psu and fan bays - testing to see if it improves cooling by forcing air out over the 'busy' side of the switch (rather than straight out the back). Instead of putting wires directly into the fan-bay connector on the mainboard, I decided to mod one of the fan trays instead (my switch shipped with 2 psu and 2 fan trays). When it comes to modding the psu, I decided to cut the entire top out of the (plastic/paper?) 'shield/cover' - this was also done to improve airflow.

All credit goes to 'dodgy route'!!! If you are interested in modding your switch, I highly recommend reading their post and checking out their google photos album (they took photos of the whole process). Click here to view their post

6610-fan-mod.jpg

After a couple of hours idling, the temperatures look good! (ambient room temp 25*c).

Code:
ICX6610-48P-Router>show chassis
The stack unit 1 chassis info:

Power supply 1 not present
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  T62
        Firmware Ver:    B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 not present
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 38.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->84       deg-C
                Speed 2:       79<-----> 87 deg-C (shutdown)

Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 31.0 deg-C
MAC 2 Temperature Readings:
        Current temperature : 38.0 deg-C
CPU Temperature Readings:
        Current temperature : 33.0 deg-C
sensor A Temperature Readings:
        Current temperature : 21.5 deg-C
sensor B Temperature Readings:
        Current temperature : 28.5 deg-C
sensor C Temperature Readings:
        Current temperature : 28.5 deg-C
stacking card Temperature Readings:
        Current temperature : 37.0 deg-C
        Warning level.......: 77.0 deg-C
        Shutdown level......: 87.0 deg-C
 

mshook

New Member
Jun 9, 2020
6
5
3
Thank you 'dodgy route' for the post that inspired me to fan-mod my icx6610 48x poe, and thank you foheesha for helping me get my switch licensed.
...
That is just brilliant. I should do that with my ICX6450, the only issue is the cover also covers the sides, so I'd have to do acrylic for side panels as well.
 
  • Like
Reactions: ip64.uk

ip64.uk

New Member
Mar 22, 2020
2
5
3
How would you rate the noise before vs after?
In my opinion, before it was way too loud for 24/7 (if in the same room).. Now it's no louder than my desktop computer.. but it really depends on how sensitive you are to noise - but without a doubt, it's a massive improvement over the stock fans (in both noise output and cooling performance). The good thing about large fans, is that they move alot of air at lower rpms - if you do hear them, its usually a 'low' frequency sound rather than an annoying high pitched one. Must also remember, I have the 48 port version with POE, so this switch might be louder than other models
 

ArmedAviator

Member
May 16, 2020
91
54
18
Kansas
Had my first real problem with my ICX6610 last night.

After setting up 2 ipv6 access-lists with about 40 ACLs each and assigning them to their respective ves, the switch rebooted itself after about 3 minutes. When it came back online, it did it again (same config). Then it did it again. No errors in the console, just a sudden reset.

I was able to login and remove the ACLs quick enough on the 4th reboot and it's been fine since.

Has anyone else noticed any reliability issues with IPv6 stuff? It seems to be that the 8.0.30 train is fairly immature with regards to IPv6.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,616
2,810
113
32
fohdeesha.com
Had my first real problem with my ICX6610 last night.

After setting up 2 ipv6 access-lists with about 40 ACLs each and assigning them to their respective ves, the switch rebooted itself after about 3 minutes. When it came back online, it did it again (same config). Then it did it again. No errors in the console, just a sudden reset.

I was able to login and remove the ACLs quick enough on the 4th reboot and it's been fine since.

Has anyone else noticed any reliability issues with IPv6 stuff? It seems to be that the 8.0.30 train is fairly immature with regards to IPv6.
well that's wild. Never seen that before, I have a bunch of v6 ACLs on a stack of 2 6610's here at home and never ran into any issues. Could you post the problematic config, and what firmware version are you on
 

HaxSmash

New Member
Oct 15, 2020
8
2
3
I'm hoping for some help. I'm a networking newbie, and after fighting with

it a few hours, I'm just not sure what to do. If anything I get the feeling that what I am trying to do is probably just dumb to begin with, so there is that.

I am hoping to be able to utilize layer 3 switching on my 6450, and use my existing Ubnt USG as my internet firewall / router.

What I have setup currently for testing is as follows.

On the USG, I have 2 new networks defined on the second LAN interface.
- VLAN 99 192.168.99.0/30 with the USG on 192.168.99.2. This is acting as my fallback network for the times that I inevitably break my SSH connection to help prevent me from having to venture to the basement with a console cable.
- VLAN 25, 192.168.25.0/24 USG listening on 192.168.25.2, DHCP server enabled, default gateway setup for 192.168.25.1

On the 6450:

I have the link between the USG and the 6450 on 1/1/1. This is tagged in VLAN 25 and VLAN99.
I have port 1/1/2 untagged in vlan 25, this is where i'm connecting my test machine. I have a helper-address set to the USG gateway for DHCP.
I have the default gateway set to 192.168.25.2

Code:
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 25 by port
 tagged ethe 1/1/1
 untagged ethe 1/1/2
 router-interface ve 25
 
vlan 99 by port
 tagged ethe 1/1/1
 router-interface ve 99

!
interface ve 25
 ip address 192.168.25.1 255.255.255.0
 ip helper-address 1 192.168.25.2
!

interface ve 99
 ip address 192.168.99.2 255.255.255.0
!

ip dns server-address 192.168.25.2
ip route 0.0.0.0/0 192.168.25.2
The issue I am having is, when configured like this, only one of my VLAN's on the 6450 is able to route outside the switch at any given point in time. Configured as shown above, my test device is able to access the internet, and the rest of my network (on the other USG lan port). DHCP works as expected, all is wonderful.

However, my VLAN99 can't access anything. For example, if I ping google.com source 192.168.99.2, I just get request timeouts. It is the same for anything I try to do when sourcing from the 99.2 address. If I switch the default gateway of the switch to be 192.168.99.1, my VLAN 99 is able to communicate to the outside world, but VLAN25 becomes effectively dead.

I'm just not sure how to configure things to have more than one VLAN functioning on my switch while using layer 3 switching. If I just use my USG as the default gateway for my laptop, everything works perfectly fine. My question is... am I trying to do something stupid? My reasoning for trying to do this is that I want to be able to do 10G switching across VLANs within my network and not be bottle-necked by the USG's 1gbe connection. I also want to continue using as much as the unifi stuff as I can, since it makes it easy to deal with IP reservations, and monitor traffic.
 

infoMatt

Active Member
Apr 16, 2019
222
99
28
I have a helper-address set to the USG gateway for DHCP.
There's no need for an helper address if the DHCP server is on the same broadcast domain (ie. the same VLAN).

if I ping google.com source 192.168.99.2, I just get request timeouts. It is the same for anything I try to do when sourcing from the 99.2 address.
That might be normal because the network 192.168.99.0/24 is locally connected on the USG, so it would forward it directly and not via 192.168.25.1, so if you've defined some policies that accept traffic from that address, those won't be matched.

Try with a traceroute, you'll see all the hops taken by a packet.

It's really a bad idea to mix up routing and switching; a routed network must not be defined on any upstream router interface, only in its route table.
 

HaxSmash

New Member
Oct 15, 2020
8
2
3
It's really a bad idea to mix up routing and switching; a routed network must not be defined on any upstream router interface, only in its route table.
Ok so that goes along with my gut feeling that I'm trying to do something that I shouldn't, my ignorance showing.

So If I'm following my options are:

a) Continue to use the USG to define all my vlans, and just use the brocade in more of a dumb layer 2 mode?

b) Scrap the vlans and just use one big flat dumb network (aka give in to my lack of knowledge)

b) Make the USG less aware of the downstream switching, define everything on the brocade except for a simple uplink / default route? I guess i'm missing something here around where the NAT happens and the routing. So much to learn.

I started writing this reply thinking I knew a path forward just to realize I'm further away than I thought. I'll have to search some more for some examples.
 

infoMatt

Active Member
Apr 16, 2019
222
99
28
my gut feeling that I'm trying to do something that I shouldn't, my ignorance showing.
Don't worry too much, everyone has learned the lessons the hard way by breaking something... at worst case, simply don't save to startup config and pull the plug :p

I guess i'm missing something here around where the NAT happens and the routing. So much to learn.
Now I am the one missing someting as I don't know about how the USG handles NAT for networks not defined... I don't think it goes too well.

A trick for learn without many trips to the basement is, provided you have a PC capable of vlan tagging, define the "transit network" (192.168.25.0/24) as a tagged VLAN on the interface "to the desk" (or even better, to another interface if you have two wires), define the "LAN" 192.168.99.0/24 inside the USG but do not assign it to the interface, and add a route for 192.168.99.0/24 via 192.168.25.1... everything """should""" work. If this is the case, grow as you like; if not, switch on the tagged VLAN and you can reach the interwebs and the configuration consoles.
Oh, and remember to define the DHCP helper on ve 99, otherwise you won't obtain a lease.

I started writing this reply thinking I knew a path forward just to realize I'm further away than I thought. I'll have to search some more for some examples.
Don't feel too bad, the hard trick is to learn it the first time, every other network is really a "rinse and repeat" situation.
The controller thingy of the UniFI world, in my opinion, does more harm than good in those situations, as it hides from sight the plain basic things, in a "let's simplify for the masses" way that I feel completely dumb, but oh-well...

Don't give up, you'll have to break stuff to learn how it works, don't be ashamed of it.