Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
You can absolutely use a single regular qsfp cable to connect the breakout port to another switch (assuming the other switches port is configured for breakout / 4x 10gbe). You can even connect a breakout port with a qsfp cable to a non breakout port on something, but only one lane will link up so it will run at 10gbps

Both 40gbe and 4x 10gbe breakout qsfp+ are electrically identical, it's 4x 10gbps lanes. Difference is in how the ASIC/PHY decides to split them (or mux them)

Hey @fohdeesha and @NablaSquaredG,

You two are champions in the ICX 6610 space and I'm running into a wall.

I have 3 6610s, stacked. I want a ring topology for the three of them.

I have an OPNSense router connected to 1/2/1 at 40G (I'd like to keep this here because my card doesn't mux). I have a Cisco N3K connected to 3/2/7-3/2/10 (it muxes to 40G). I have a Mellanox SX6036 connected to 3/2/1 at 40G (I'd like to keep this connection here because I don't know how to create a setting on the SX6036 to mux 4x10G lanes).

My currently linear topology is: 1/2/6 -> 2/2/6, 2/2/1 -> 3/2/1... I would like to connect 3/2/2-3/2/5 -> 1/2/2-1/2/5. My cable is: https://www.amazon.com/gp/product/B01N4GKPQ8/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1

I'm a bit new to networking, so I'm pretty ignorant. Long story short, it seems no matter what I do, I can't get the stack to move to a ring topology when using the connection 3/2/2-3/2/5 -> 1/2/2-1/2/5.

I'd frankly prefer to use all of the breakout ports for my ring topology and save the pure 40G ports for other things, but I couldn't get the stack to configure on the breakout ports. Do either of you know how to help my sad/ignorant self?
 
Last edited:
I'd frankly prefer to use all of the breakout ports for my ring topology and save the pure 40G ports for other things, but I couldn't get the stack to configure on the breakout ports. Do either of you know how to help my sad/ignorant self?

I remember having a similar issue a while back.

According to the manual, you can use a "ring stack topologies with partial cabling" - however it seems like one 40G port for stacking is always required. I'm also not sure whether you deconfigure the stack-trunk (which is the thing that allows using the ports as data ports) and use partial cabling at the same time.

If I remember correctly, I used 40G for stacking and configured a 4x10G trunk to the SX6036 with 4x10G link aggregation on the SX6036. It's not that difficult
 
After the...longest...time, I think it's time to upgrade. :) Ran out of all 10g/40g ports on my 6610...need more (adding some more production capacity).

Sanity check - Arista 7050QX-32S still the most viable option? (has more than enough 10g/40g ports. Would have loved some 2.5/25gb ports as well, but it's not critical)

Edit: Anybody interested in a modded 6610 with 3x 120mm fans on top (hacked up top cover) and NO PSU fans (PWM trickery...)? Been running stable for years...
 
just a heads up, the icx7150-24p/48p (and non-P variants!) can be had for <$200 on ebay these days, and these are arguably a contender for the "go-to" homelab switch. You can run it fanless (no need for any mods, just a simple CLI command) if you are okay with 150 watts of PoE output. They also sip power (15ish watts for non-P and 30-ish for P)
 
Sanity check - Arista 7050QX-32S still the most viable option? (has more than enough 10g/40g ports. Would have loved some 2.5/25gb ports as well, but it's not critical)
7050QX is quite different from the ICX6610? The 7050QX is Fiber only.

The QX-32S (S-version) runs some slightly newer EOS version and has 4 additional SFP ports. The QX-32 (without S) runs an older EOS and lacks the SFP ports. Otherwise they are identical. The non-S models are a lot cheaper, usually.
 
7050QX is quite different from the ICX6610? The 7050QX is Fiber only.

The QX-32S (S-version) runs some slightly newer EOS version and has 4 additional SFP ports. The QX-32 (without S) runs an older EOS and lacks the SFP ports. Otherwise they are identical. The non-S models are a lot cheaper, usually.
I need very few RJ-45 ports. Most everything is fiber in my rack (DACs), either 10g or 40g. The S version of the Arista is appealing, not necessarily for the extra SFP ports, but because of the ability to run newer versions of EOS.

That said, I'm not married to this idea, I can be swayed. My core requirements would be at least 24x 10gb SFPs and 6x 40gb SFPs. Everything else is negotiable.

Now, there's of course the Mellanox SX6036 as well...but gotta go through some contortions to get it working right.
 
just a heads up, the icx7150-24p/48p (and non-P variants!) can be had for <$200 on ebay these days, and these are arguably a contender for the "go-to" homelab switch. You can run it fanless (no need for any mods, just a simple CLI command) if you are okay with 150 watts of PoE output. They also sip power (15ish watts for non-P and 30-ish for P)
This is not for a homelab per se... :) Yes, it will run in my home, but I run my production business infra from home (and some colo).
 
I remember having a similar issue a while back.

According to the manual, you can use a "ring stack topologies with partial cabling" - however it seems like one 40G port for stacking is always required. I'm also not sure whether you deconfigure the stack-trunk (which is the thing that allows using the ports as data ports) and use partial cabling at the same time.

If I remember correctly, I used 40G for stacking and configured a 4x10G trunk to the SX6036 with 4x10G link aggregation on the SX6036. It's not that difficult

I saw somewhere on reddit that Fohdeesha had helped someone do this by using, as you're saying, one of the 40G stacking ports and trunking the breakout ports. But when I tried to reconfigure it (stack-trunk 1/2/6 to 1/2/7) it doesn't show as a ring
I remember having a similar issue a while back.

According to the manual, you can use a "ring stack topologies with partial cabling" - however it seems like one 40G port for stacking is always required. I'm also not sure whether you deconfigure the stack-trunk (which is the thing that allows using the ports as data ports) and use partial cabling at the same time.

If I remember correctly, I used 40G for stacking and configured a 4x10G trunk to the SX6036 with 4x10G link aggregation on the SX6036. It's not that difficult

So there's a clever idea (referring to your doing LACP on the SX6036 to get the 40G from a breakout on the ICX-6610). Can I LACP the breakout ports on Stack Unit 1 and Stack Unit 3? And would they then stack? I'll look into doing LACP on the breakout ports.

Also, I read somewhere (on reddit) that someone was using the MCX354A on one of the breakout ports but still getting 40G (although, I think he might have been using a stack-trunk and that 40G was probably from his 1/2/1 or 1/2/6). So maybe that's an option for me, too (to move that connection 1/2/2 or 1/2/7 and then connect the stack to 1/2/6 and 3/2/1).

When I tried to use "stack unit 3" -> stack-trunk 3/2/6 to 3/2/7 it didn't show as a ring topology. It showed as a linear topology with a trunk on the connection between stack unit 2 and stack unit 3 (even though the cables were physically connected from stack unit 3 to stack unit 1).
 
Can I LACP the breakout ports on Stack Unit 1 and Stack Unit 3? And would they then stack? I'll look into doing LACP on the breakout ports.
Sure, LAG across the stack is possible (basically the alternative to MLAG / MCT)

Also, I read somewhere (on reddit) that someone was using the MCX354A on one of the breakout ports but still getting 40G (although, I think he might have been using a stack-trunk and that 40G was probably from his 1/2/1 or 1/2/6).
Certainly not. That is not possible. It might have linked, but not with 40G.
 
That said, I'm not married to this idea, I can be swayed. My core requirements would be at least 24x 10gb SFPs and 6x 40gb SFPs. Everything else is negotiable.
Very much depends on how important power consumption is for you.
If low power consumption is critical: Mellanox SX1024 (SFP+QSFP) or SX6036 (QSFP only)

If not: Arista 7050SX-72Q (SFP + QSFP) or 7050QX-32(S) (QSFP only, except the 4 SFPs on the -S)

The Aristas are generally nicer than the SX6036 software-wise - the latest available MLNX-OS on the SX6036 is ANCIENT, has some bugs and an ANNOYINGLY SLOW control plane.
 
The Aristas are generally nicer than the SX6036 software-wise - the latest available MLNX-OS on the SX6036 is ANCIENT, has some bugs and an ANNOYINGLY SLOW control plane.
Aware of that. The low power consumption is appealing though. (I mean, how often do I mess with the control plane? Not much, after initial setup)
 
If not: Arista 7050SX-72Q (SFP + QSFP) or 7050QX-32(S) (QSFP only, except the 4 SFPs on the -S)

QSFP that you can breakout to 4x10G sfp links which is what I used to do. Either dac or get optics which will need MTP fiber cable to 4 LC. Word of warning if you leave the fans on auto the 7050QX min fan is 60% which you can overwrite and I ran it at 35-40% depending on the time of year and even then it runs the asic around 50c. Just so you know. It is an awesome switch.
 
Now, there's of course the Mellanox SX6036 as well...but gotta go through some contortions to get it working right.
Be careful with this one as only some of the ports can be broken out to 4x10 and not all of them (or majority) so you have to make sure you read the documentation, have one of these as well :)
 
Thank you both for trying to help. Seems I have bigger problems as I came back to a dead switch. Appears the power supply toasted on my core switch (burning smell). So I've had to put back my old Cisco switch for now. I'll explore my options for replacement after the holidays.

Did an autopsy on the switch (6450-24P) and there's a hole burned in one of the chips on what I'm assuming is the PoE board, and through the other side of the board. I imagine something like this would have blown the fuse on the power supply assuming there is one. I didn't explore any further after finding this catastrophic failure.

Not sure I want to replace it with another Brocade considering I only got about six months out of it. I've never had a Cisco die on me. It's getting harder to use anything but their really ancient stuff for home lab use because of their unfriendly licensing requirements however.
 

Attachments

  • board.jpg
    board.jpg
    489.5 KB · Views: 31
  • chip.jpg
    chip.jpg
    523.7 KB · Views: 30
The QX-32S (S-version) [...] has 4 additional SFP ports.
Unlike the 7060CX-32S and later generations, the 7050X's SFP+ ports are just a convenience breakout. The 7050QX-32S's are tied to port 5. Either the SFP+ ports or QSFP+ port is enabled at any given time.


Can I choose which QSFP+ interface to disable in lieu of the SFP+ ports on the DCS-7050QX-32S?​
The leftmost QSFP+ port (Et5/1) must be disabled, as this shares the connections with the 4 SFP+ ports.​
​
How do I enable/disable the SFP+ ports on 7050QX-32S?​
The mode of the first SFP+ and QSFP+ ports is selected as follows:​
Switch(config)#hardware port-group 1 select Et1-4 [SFP+]​
Switch(config)#hardware port-group 1 select Et5/1-4 [QSFP+]​
 
Did an autopsy on the switch (6450-24P) and there's a hole burned in one of the chips on what I'm assuming is the PoE board, and through the other side of the board. I imagine something like this would have blown the fuse on the power supply assuming there is one. I didn't explore any further after finding this catastrophic failure.
Classic failure mode on the Brocade / Ruckus PoE switches...
 
Mellanox SX6036 and various odds and ends (cables, optics etc) ordered. We'll see how it plays out.
 
Looking for some help from people here (esp @kapone you've been amazing in the past!).

I've gotten the standard Opnsense and ICX6610 setup working where all my local traffic between my 3 VLANs are routed internally, anything going to the internet is routed to OpnSense.

However, I just got a Reolink doorbell and very annoyingly, it will not allow local connections unless they appear to come from the same VLAN. The Reolink doorbell is on the IOT VLAN, and client devices connecting to the doorbell are usually on the "STANDARD" or "ADMIN" VLANs.

On the Reolink forums, the solution given was to Source NAT the client IP so that it appears to be coming from the same VLAN as the doorbell.

ICX switches cannot SNAT. So conceptually i think i have to route any traffic from the STANDARD or ADMIN VLANs directed to the Reolink Doorbell IP out to Opnsense for Opnsense to SNAT and send back into the IOT VLAN.

Before I start messing around blindly, I'm guessing i'd have to add an entry to the ICX6610 routing table and also some routing rules/SNAT rules in Opnsense.
Not sure of the exact details.

If someone could guide me on routing table modifications in the ICX6610 to achieve the above that would be great!

Thanks!!
 
Well, i seem to lack a basic understanding of routing tables.

I added a static route for anything destined for the Reolink IP to be sent to Opnsense. Of course that sent anything destined for the Reolink back to Opnsense in an infinite loop, duh.

Is it possible to add a source - destination to the ICX routing tables? or is it just destination only?