Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

crackelf

Member
Apr 11, 2021
74
6
8
Quick sanity check:

ICX 7450 with 2x 40G modules and stack disabled. Can I use Mellanox ConnectX-4 cards in 40G ethernet mode with these? Was planning on either some DACs or MTP-MTP with optics on either end.

Wanted to check before pulling the trigger on all these. Thanks!
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Quick sanity check:

ICX 7450 with 2x 40G modules and stack disabled. Can I use Mellanox ConnectX-4 cards in 40G ethernet mode with these? Was planning on either some DACs or MTP-MTP with optics on either end.

Wanted to check before pulling the trigger on all these. Thanks!
yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Quick sanity check:

ICX 7450 with 2x 40G modules and stack disabled. Can I use Mellanox ConnectX-4 cards in 40G ethernet mode with these? Was planning on either some DACs or MTP-MTP with optics on either end.

Wanted to check before pulling the trigger on all these. Thanks!
On the ICX, the ports are just ports. Optics are probably safest; I don't know if they QSFP on a 7450 can understand a QSFP28 (is that what the CX4 is?).

On my 7450 I'm using both of them as a LAG to my Arista 7050 core.
 
  • Like
Reactions: crackelf

crackelf

Member
Apr 11, 2021
74
6
8
yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay
Thanks for the response and the link! I had been looking at these Brocade optics on eBay and was going to try out these Arista MTP-MTP 12 fibers OM4 (also eBay), but I much prefer the idea of cheap LC. I really appreciate your guidance on this! Will post back when everything arrives.

Last potential gotcha: is it possible to fall back to 10G with a 40G <---> 10G breakout cable between card <---> switch if the QSFP ports on the switch fail for whatever reason, or would a QSFP to SFP+ Transceiver Adapter make more sense?

On the ICX, the ports are just ports. Optics are probably safest; I don't know if they QSFP on a 7450 can understand a QSFP28 (is that what the CX4 is?).

On my 7450 I'm using both of them as a LAG to my Arista 7050 core.
I thought that made the most sense, but after living through various licensing hells I was spooked somehow they wouldn't allow them to be used as non-stacking ports. Thanks for the reassurance. I actually have no idea myself if the 7450 can handle the QSFP28, but I'll report back with my findings even if this all blows up in my face spectacularly.
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Last potential gotcha: is it possible to fall back to 10G with a 40G <---> 10G breakout cable between card <---> switch if the QSFP ports on the switch fail for whatever reason, or would a QSFP to SFP+ Transceiver Adapter make more sense?
The ICX7450 does not support breakout. If you want 10G, you have to use the 4x10g module.

The BiDi posted by fohdeesha is known to work and is an excellent cheap choice. I don't think those 16Gb FC optics would work the way you want, assuming the switch can recognize them at all.

Edit: maybe that optic would work, it's 4x 16 Gb lanes. The BiDi is still cheaper at $12 vs $40.
 

crackelf

Member
Apr 11, 2021
74
6
8
The ICX7450 does not support breakout. If you want 10G, you have to use the 4x10g module.

The BiDi posted by fohdeesha is known to work and is an excellent cheap choice. I don't think those 16Gb FC optics would work the way you want, assuming the switch can recognize them at all.

Edit: maybe that optic would work, it's 4x 16 Gb lanes. The BiDi is still cheaper at $12 vs $40.
Sorry I probably didn't explain this the right way - I'm saying the breakout goes from card to switch so the QSFP side plugged into the Mellanox and 1 of the 10G plugged into the 4x10g module, but the more I type this out the more I realize the adapter is the way to go here. I completely believe that the breakout doesn't work from the switch side since it doesn't say anywhere in the software or on the hardware that breakout is doable.

I posted said Brocade optics because another thread on here (which I now for the life of me can't find) reporting them as functional with the 6610, but we have been blessed by now both fohdeesha himself & LodeRunner with better recommendations, so thanks for straightening me out! I barely skated by setting up my last 10g topology with a 7150, so my hilarious lack of fiber knowledge is really shining here.
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Ah. The ICX would be expecting a LAG at that point, so if the card supports being broken out and then you LAG it in the OS, then I expect it would work.
 

acurax04

New Member
Feb 18, 2022
2
1
3
it sounds like you haven't actually plugged the switch into your network? The usb-c serial connection is not a network connection, as the guide states you need to connect the switches dedicated management ethernet port (NOT any of the regular ports) to your network
:oops: Thank you! I was able to complete the guide--appreciate this community.
 
  • Like
Reactions: fohdeesha

kilaketia

New Member
Mar 18, 2020
2
0
1
Hi, I bought a ICX6450-24P and I tried using the console port to configure it but it's dead silent. I bought this cable https://www.amazon.fr/gp/product/B01N1625DE/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1, I've tried another cable that comes with Zyxel router (simple RJ45 to DB9) with my server or another server to try to communicate with the switch. But no mater what I've tried it's stays silent. I can access it with telnet but no password is configured, http but no login combination works, same for ssh. I reset it multiple times...

Does anyone has an idea to help me ?
 

kilaketia

New Member
Mar 18, 2020
2
0
1
Alright I still don't have access via the console port, but I found a way to access privileged exec mode via telnet by using "enable system-monitoring all" instead of just enable...

It's on v8030t.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Sorry I probably didn't explain this the right way - I'm saying the breakout goes from card to switch so the QSFP side plugged into the Mellanox and 1 of the 10G plugged into the 4x10g module, but the more I type this out the more I realize the adapter is the way to go here. I completely believe that the breakout doesn't work from the switch side since it doesn't say anywhere in the software or on the hardware that breakout is doable.

I posted said Brocade optics because another thread on here (which I now for the life of me can't find) reporting them as functional with the 6610, but we have been blessed by now both fohdeesha himself & LodeRunner with better recommendations, so thanks for straightening me out! I barely skated by setting up my last 10g topology with a 7150, so my hilarious lack of fiber knowledge is really shining here.

mellanoxeseses (and most NICs) don't support qsfp breakout, it's a single connection of either 40gbe or 10gbe. If you want to drop it to a 10gbE SFP connection, search ebay for qsfp > sfp adapter and stick that in the NIC - Mellanox MAM1Q00A-QSA 655874-B21 40G QSFP+ To 10G SFP+ Network Cable Adapter | eBay
 

jcstill

New Member
Aug 10, 2021
4
0
1
Southern California
can you try to run that again (mainly the update u boot command), then reset the switch, (just send the command "reset" but be sure to stop it back into the bootloader again (if it boots all the way into the OS, it may downgrade uboot again). once you stop it back in uboot after the reboot, it should hopefully be in the new version of uboot, then try the update primary command again
I ran the u boot update again and reset the switch. The version is now 10.1.18. I ran into the same issue for the primary update where I'd get the "Primary image download failed". I think this had to do with my tftp server timing out. Rather than spending hours trying to diagnose, I just used tftp64.exe on a windows box and now am on v08.0.90 updating to 08.0.95.

Thanks for the help!
 

-MoNsTeRRR

New Member
May 24, 2020
5
1
3
Hey everyone,
Is anyone here running "Oxidized" on the ICX 6610 to backup the config ? I am have issues with getting it run on this switch and so I thought I would ask here on the thread where I have learned so much about this device.

I have "Oxidized" running on Librenms.

Thanks.
Hello i've the same issue as your :).

I'm using oxidized (not the librenms plugin) i'm running the latest release from oxidized (git install)
Code:
git show --summary
commit 4a4d0c4730700ff219f8af7710adac2973c827ef (HEAD -> master, origin/master, origin/HEAD)
I've added this command to bypass enable issue
Code:
aaa authentication enable implicit-user
I'm stuck at this point :
Code:
[...]
D, [2022-02-20T11:48:25.518942 #236703] DEBUG -- : lib/oxidized/input/cli.rb Running pre_logout commands at 192.168.0.1
D, [2022-02-20T11:48:25.519219 #236703] DEBUG -- : lib/oxidized/input/ssh.rb exit @ 192.168.0.1 with expect: nil
D, [2022-02-20T11:48:26.226598 #236703] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2022-02-20T11:48:27.227918 #236703] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
[...]
W, [2022-02-20T11:52:30.191710 #236703]  WARN -- : 192.168.0.1 raised Timeout::Error with msg "execution expired"
D, [2022-02-20T11:52:30.191783 #236703] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH failed for 192.168.0.1
D, [2022-02-20T11:52:30.191852 #236703] DEBUG -- : lib/oxidized/job.rb: Config fetched for 192.168.0.1 at 2022-02-20 10:52:30 UTC
W, [2022-02-20T11:52:30.508846 #236703]  WARN -- : /192.168.0.1 status no_connection, retry attempt 1
D, [2022-02-20T11:52:30.509142 #236703] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 1 of 2
D, [2022-02-20T11:52:30.509382 #236703] DEBUG -- : lib/oxidized/worker.rb: Added /192.168.0.1 to the job queue
As I see on github there as been an issue and you need to run two times the exit command as I see in lib/oxidized/model/fastiron.rb the pre_logout commands as two exit so I don't know what it's going on.

Does anyone know how to handle it ?
 

adman_c

Active Member
Feb 14, 2016
257
135
43
Chicago
Does anyone here have their WAN run through their switch (vlan isolated) to the router and then back? I’m trying out t-mobile 5g as a failover WAN, and my networking ”closet” is on the ground floor of my house. Unsurprisingly, cellular signals are much better on the top floor of my house. I have a network drop up there but no great way to connect directly from that to my pfsense router without going through the switch. Is it possible (and safe) to have a WAN_Transit VLAN that I can use to connect the t-mobile modem on the top floor through my ICX6450 to the second WAN port on my pfsense box? Thanks!
 

kpfleming

Active Member
Dec 28, 2021
383
205
43
Pelham NY USA
Does anyone here have their WAN run through their switch (vlan isolated) to the router and then back? I’m trying out t-mobile 5g as a failover WAN, and my networking ”closet” is on the ground floor of my house. Unsurprisingly, cellular signals are much better on the top floor of my house. I have a network drop up there but no great way to connect directly from that to my pfsense router without going through the switch. Is it possible (and safe) to have a WAN_Transit VLAN that I can use to connect the t-mobile modem on the top floor through my ICX6450 to the second WAN port on my pfsense box? Thanks!
Sure, both possible and 'safe'. It's just a layer 2 connection, should work fine. if you can put a regular unmanaged switch between the modem and the pfsense machine, you can put a fancy managed switch there too :)
 
  • Like
Reactions: adman_c

adman_c

Active Member
Feb 14, 2016
257
135
43
Chicago
Sure, both possible and 'safe'. It's just a layer 2 connection, should work fine. if you can put a regular unmanaged switch between the modem and the pfsense machine, you can put a fancy managed switch there too :)
I guess I’m wondering how worried I need be about a VLAN hopping attack by having a WAN connection pass through my primary switch before any firewalling happens.
 

Rttg

Member
May 21, 2020
71
47
18
I guess I’m wondering how worried I need be about a VLAN hopping attack
That be a major vulnerability in a fairly widely deployed switch - not impossible but highly unlikely. I only moonlight as a network engineer in my homelab, but I’d wager you’re pretty safe.
 
  • Like
Reactions: adman_c

ccie4526

Member
Jan 25, 2021
88
56
18
I guess I’m wondering how worried I need be about a VLAN hopping attack by having a WAN connection pass through my primary switch before any firewalling happens.
VLAN hopping attacks will normally only jump you off of whatever tagged VLAN you're on over to an untagged VLAN. If you don't have an untagged VLAN configured on your trunk links, then attack mitigated. Even better if you do NOT use VLAN 1 anywhere in your network.

I run internet outside on my VLAN 999, and my internal VLANs are using other tag numbers, and NOT VLAN 0001.

VLAN hopping attacks succeed because of either unmanaged (read: not VLAN-aware) switches, or use of VLAN 1. Don't do either of those.

And yeah, I moonlight as a network engineer at the world's largest beer brewery. And it's not the world's largest beer brewer. ;-)
 
  • Like
Reactions: adman_c

Dave Corder

Active Member
Dec 21, 2015
290
184
43
41
For a WAN transit VLAN, I'd be more concerned about the switch sending management/discovery frames and other "junk" to the T-Mobile router (and maybe having that device "lock onto" the switch's MAC address instead of your router's). I have a vague recollection of that sort of thing being discussed somewhere in this thread in the past...
 
  • Like
Reactions: adman_c