Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[NateR]

Hello all,

I've got an ICX6450-24p I've had in use for a year or so now. I've noticed that when I mess around with stuff in my rack, sometimes I'll lose the SFP ports (no status lights), which I use for uplink to the upstream switch, requiring a reboot of the switch. I believe I've finally figured out what is causing it - I disconnected a shielded network cable the other day, and happened to touch the metal shielding on the connector and shocked it with ESD (it's pretty dry here, and I typically touch the rack before anything else to discharge it, but either forgot to or just moving around built up a charge). The SFP ports went dark when this happened. Rebooted, and they came back up.

Any ideas on how to keep this from repeatedly happening?

Also, has anyone found a good alternative to OEM rack-mount ears? Mine came with a couple, but as I didn't have a rack at the time, I stupidly removed them and put them "where I will remember them..." well, after much searching, I have failed to find them.

Thanks

 

[dswartz]

Yeah... I like Mellanoxes approach, where you can have at least images downloaded on the switch, but not installed (just ready to install).

This doesn't work from the bootloader though, but that would be a great idea in case some switch manufacturer reads this

You beat me to it. Yeah, you need to be up and running to do that, but both flash partitions were borked. "Can't get there from here". What I ended up doing:

1. Login to 7250 stack with my iphone and delete the LAG.
2. Unplug one of the LAG cables at my end to avoid STP issues.
3. Move the other cable to the mgmt port on the 7150.
4. Boot from tftp on the VM specifying the old FW.
5. Copy from tftp on the VM with old FW to primary.
6. Reboot from primary.
7. Set up minimal config so I can http in and finish config (there wasn't much unlike the 7250 stack).
8. copy from primary flash to secondary flash.
9. Login to 7250 stack and reinstate the LAG.
10. Restore cables to original positions.
11. (minimize future hassles) Install tftpd-hpa on a 1U rackmount linux host by my desktop that is a secondary backup target.
12. Bitch-slap myself for getting into this cluster-f*ck to begin with LOL.

 

[dswartz]

Hello all,

I've got an ICX6450-24p I've had in use for a year or so now. I've noticed that when I mess around with stuff in my rack, sometimes I'll lose the SFP ports (no status lights), which I use for uplink to the upstream switch, requiring a reboot of the switch. I believe I've finally figured out what is causing it - I disconnected a shielded network cable the other day, and happened to touch the metal shielding on the connector and shocked it with ESD (it's pretty dry here, and I typically touch the rack before anything else to discharge it, but either forgot to or just moving around built up a charge). The SFP ports went dark when this happened. Rebooted, and they came back up.

Any ideas on how to keep this from repeatedly happening?

Also, has anyone found a good alternative to OEM rack-mount ears? Mine came with a couple, but as I didn't have a rack at the time, I stupidly removed them and put them "where I will remember them..." well, after much searching, I have failed to find them.

Thanks

Wish I could help. My grumble about rackmount stuff: the nuts everyone provides have the same outside square size to fit the rack posts, but I've found that when you order a set of nuts and bolts, the nuts and bolts are not always the exact right size between different vendors. e.g. sometimes a bolt from vendor A is a bit too big to fit in the hole of a nut from vendor B. Other times, it's a bit too small, and doesn't engage the threads (but just slides right in, which is obviously useless).

 

[dswartz]

Weirdly I had something similar when my area had major power outages from an ice storm that took power out for hours, exhausted the UPS and then proceeded to flicker on and off for several more hours. Although I still had a switch that seemed to function correctly, I could neither SSH into it or do anything directly at the console. Ultimately pulling the plug and then powering back up fixed that glitch, though I was nervous and ready to reinstall on a backup switch.

I had something very similar happen!

 

[dswartz]

what da heck is an icx6150? lol

I don't know of any low level commands that completely bypass physical SFP presence, but from the page you linked it looks like you should just solder pin 6 to ground on the optic itself to solve it

the icx6150 is the tofu version of a beefy switch

 

[Shad]

What I'd like to do next is replace one of the switches with the PoE version, ICX6540-48P. The documentation and video guides say that as long as the replacement stack member is the same model and has been erased, it will be adopted and the config applied. "It just works".

Does anybody know if that is also the case when replacing non-PoE with PoE version of otherwise the exact same model?

Any thoughts on this one? I'm going to attempt it tomorrow now I have a replacement switch sorted out with updated firmware, a license, and empty config.

 

[tangofan]

Just wanted to drop in and say a special "thank you" to @fohdeesha Got my used ICX7150-24p up and running using your instructions a bit over 2 years ago. Just upgraded firmware to 8095h and everything went as smooth as butter.

A bit of beer money is heading your way. Thanks again for all the good work you do here.

 

[fohdeesha]

Just wanted to drop in and say a special "thank you" to @fohdeesha Got my used ICX7150-24p up and running using your instructions a bit over 2 years ago. Just upgraded firmware to 8095h and everything went as smooth as butter.

A bit of beer money is heading your way. Thanks again for all the good work you do here.

Thank you for the donation!

 

[Shad]

What I'd like to do next is replace one of the switches with the PoE version, ICX6540-48P. The documentation and video guides say that as long as the replacement stack member is the same model and has been erased, it will be adopted and the config applied. "It just works".

Does anybody know if that is also the case when replacing non-PoE with PoE version of otherwise the exact same model?

In case anyone finds this in the future, I played it safe. Having exported the config it was clear it wasn't going to work plug-n-play because the POE module has a different name so the stack part of the config wouldn't have matched up. I modified the config with the correct module name and restored it to the new POE switch and it was happy. Then I erased the other switch for the stack and it hooked it up, and the config automatically copied over as expected. All sorted

 

[kapone]

Thank you for the donation!

...

 

[NablaSquaredG]

I've lent an ICX6610 to a friend, config is stock (no VLANs, nothing)

He keeps telling me that the ICX6610 doesn't allow DHCP to passthrough? I have no idea, I've never had any issue like that. He's using the 4x10G port with a breakout cable and one of the 10G ports on the front. Other traffic seems to go through.

I suppose it's a Layer 8 error on his side...

 

[MikeSpears]

Having some issues with the 40G to 4x 10G breakout cable I bought (https://www.amazon.com/dp/B01LAR98YK and https://www.amazon.com/dp/B00U8LY4Y4). When I attach the cable to one of the breakout ports and the other end to my servers, either only 1 or 2 will link up, or none will link. Is this potentially a bad cable or user error?

 

[DerTom]

Hi!

I've got myself the Brocade ICX-6450-48P and think that I do have lost myself during configuration... At least I'm missing something and ask for your help.

What am I trying to achieve:
I do have configured four gateways and four vlans at a pfSense-firewall. I specified a specific gateway for each vlan using the firewall-rules. To get the inter-vlan-routing off the firewall I got the ICX-6450. As much as I understand the documentation or what I have read I have to configure one port with all four 'pfSense-vlans' being tagged and being connected to the pfSense-box. Next is to create four new 'ICX-vlans' for my network, create the router-interface for each vlan and specify the gateway for each 'ICX-vlan' by using the default-gateway option (IP of the created tagged vlan-port).

Right here I'm stuck...

The ICX is running the 'latest' router-firmware:




I do have created a vlan 72 for testing and the 've 72' interface:



There is no option to define a 'default-gateway' for the router-interface:



or the vlan itself:



Is it, that the 6450 isn't having this option as the firmware is 'too old'
or
my configuration is just wrong
or
I'm missing something else?

If there is someone kind enough to help me out, every hint is kindly appreciated.

Best wishes from Germany!

Tom

 

[itronin]

Hi!

I've got myself the Brocade ICX-6450-48P and think that I do have lost myself during configuration... At least I'm missing something and ask for your help.

What am I trying to achieve:
I do have configured four gateways and four vlans at a pfSense-firewall. I specified a specific gateway for each vlan using the firewall-rules. To get the inter-vlan-routing off the firewall I got the ICX-6450. As much as I understand the documentation or what I have read I have to configure one port with all four 'pfSense-vlans' being tagged and being connected to the pfSense-box. Next is to create four new 'ICX-vlans' for my network, create the router-interface for each vlan and specify the gateway for each 'ICX-vlan' by using the default-gateway option (IP of the created tagged vlan-port).

Right here I'm stuck...

Is this a requirement: Specific default gateway per VLAN?
Or
Are you simply as you said trying to get inter-vlan routing off the firewall? Ie. use the ICX6450 as your "core router"?

I'm going to leave this drawing here and see if it helps:

Spoiler: 4 Home VLAN Network with ICX6450

 

[rigel.local]

Hi, can someone confirm that the following license hack ONLY unlocks 10gb SFP+ ports and not 40gb ports at the back of the Brocade ICX 6610 model?

ICX6610 Licensing - Fohdeesha Docs

fohdeesha.com

How can I unlock those two 40gb ports?

 

[itronin]

How can I unlock those two 40gb ports?

the 4 QSFP ports on the back are not locked. See post #4
they are however configured by default for stacking.
this post talks about a mix of stack and not stacking qsfp ports.
You can configure all 4 QSFP for no stacking and then use them to connect to hosts e.g. 2x 40Gbps and 8 x 10Gbps.

 

[DerTom]

Is this a requirement: Specific default gateway per VLAN?
Or
Are you simply as you said trying to get inter-vlan routing off the firewall? Ie. use the ICX6450 as your "core router"?

I'm going to leave this drawing here and see if it helps:

Spoiler: 4 Home VLAN Network with ICX6450

View attachment 27640

As much as I know I do need a separate gateway for each vlan as there are different wan-gateways/purposes for each vlan.
#1 gateway vlan 71 -> default gateway wan
#2 gateway vlan 72 -> vpn
#3 gateway vlan 73 -> vpn-streaming
#4 gateway vlan 74 -> vpn-work
Having vlan99/ve99 for all four vlans will not work as I don't see a way to separate the traffic for the different gateways.

I'm thinking about a setup like:
(icx-6450 -> pfsense -> gateway)
#1 vlan71 (ve 71: 192.168.71.253, default-gateway: 192.168.171.254) -> vlan171 (192.168.171.0/24, 192.168.171.254) -> default-gateway: 10.0.71.1
#2 vlan72 (ve 72: 192.168.72.253, default-gateway: 192.168.172.254) -> vlan172 (192.168.172.0/24, 192.168.172.254) -> default-gateway: 10.0.72.1
#3...
#4....

The documentation (for configuring the default-gateway) I do see is for fastiron 08050 - a version I will not get with the 6450!?

 

[MikeSpears]

Having some issues with the 40G to 4x 10G breakout cable I bought (https://www.amazon.com/dp/B01LAR98YK and https://www.amazon.com/dp/B00U8LY4Y4). When I attach the cable to one of the breakout ports and the other end to my servers, either only 1 or 2 will link up, or none will link. Is this potentially a bad cable or user error?

I suspect I fixed this, but will know for sure later as my lab lives in a datacenter so gotta go back to test... I never disabled stacking on my switch.

 

[kpfleming]

As much as I know I do need a separate gateway for each vlan as there are different wan-gateways/purposes for each vlan.
#1 gateway vlan 71 -> default gateway wan
#2 gateway vlan 72 -> vpn
#3 gateway vlan 73 -> vpn-streaming
#4 gateway vlan 74 -> vpn-work
Having vlan99/ve99 for all four vlans will not work as I don't see a way to separate the traffic for the different gateways.

This sort of thing is what VRFs were invented for. You create a VRF for each 'virtual routing table' you need, with the proper routes to external networks, and assign VLANs to the VRFs they need.

The issue here is not the existence of different gateways, it's the existence of different routing tables.

 

[itronin]

This sort of thing is what VRFs were invented for. You create a VRF for each 'virtual routing table' you need, with the proper routes to external networks, and assign VLANs to the VRFs they need.

The issue here is not the existence of different gateways, it's the existence of different routing tables.

and @DerTom

Concur that VRF would solve this.

But, does the ICX6450 have VRF support?

AFAIK - I do not think VRF support is available on the ICX6450 (first post in the thread and here.)