Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,691
1,123
113
Sanity check - Arista 7050QX-32S still the most viable option? (has more than enough 10g/40g ports. Would have loved some 2.5/25gb ports as well, but it's not critical)
7050QX is quite different from the ICX6610? The 7050QX is Fiber only.

The QX-32S (S-version) runs some slightly newer EOS version and has 4 additional SFP ports. The QX-32 (without S) runs an older EOS and lacks the SFP ports. Otherwise they are identical. The non-S models are a lot cheaper, usually.
 

kapone

Well-Known Member
May 23, 2015
1,285
738
113
7050QX is quite different from the ICX6610? The 7050QX is Fiber only.

The QX-32S (S-version) runs some slightly newer EOS version and has 4 additional SFP ports. The QX-32 (without S) runs an older EOS and lacks the SFP ports. Otherwise they are identical. The non-S models are a lot cheaper, usually.
I need very few RJ-45 ports. Most everything is fiber in my rack (DACs), either 10g or 40g. The S version of the Arista is appealing, not necessarily for the extra SFP ports, but because of the ability to run newer versions of EOS.

That said, I'm not married to this idea, I can be swayed. My core requirements would be at least 24x 10gb SFPs and 6x 40gb SFPs. Everything else is negotiable.

Now, there's of course the Mellanox SX6036 as well...but gotta go through some contortions to get it working right.
 

kapone

Well-Known Member
May 23, 2015
1,285
738
113
just a heads up, the icx7150-24p/48p (and non-P variants!) can be had for <$200 on ebay these days, and these are arguably a contender for the "go-to" homelab switch. You can run it fanless (no need for any mods, just a simple CLI command) if you are okay with 150 watts of PoE output. They also sip power (15ish watts for non-P and 30-ish for P)
This is not for a homelab per se... :) Yes, it will run in my home, but I run my production business infra from home (and some colo).
 

SJENSEN721

New Member
Aug 25, 2024
4
1
3
I remember having a similar issue a while back.

According to the manual, you can use a "ring stack topologies with partial cabling" - however it seems like one 40G port for stacking is always required. I'm also not sure whether you deconfigure the stack-trunk (which is the thing that allows using the ports as data ports) and use partial cabling at the same time.

If I remember correctly, I used 40G for stacking and configured a 4x10G trunk to the SX6036 with 4x10G link aggregation on the SX6036. It's not that difficult
I saw somewhere on reddit that Fohdeesha had helped someone do this by using, as you're saying, one of the 40G stacking ports and trunking the breakout ports. But when I tried to reconfigure it (stack-trunk 1/2/6 to 1/2/7) it doesn't show as a ring
I remember having a similar issue a while back.

According to the manual, you can use a "ring stack topologies with partial cabling" - however it seems like one 40G port for stacking is always required. I'm also not sure whether you deconfigure the stack-trunk (which is the thing that allows using the ports as data ports) and use partial cabling at the same time.

If I remember correctly, I used 40G for stacking and configured a 4x10G trunk to the SX6036 with 4x10G link aggregation on the SX6036. It's not that difficult
So there's a clever idea (referring to your doing LACP on the SX6036 to get the 40G from a breakout on the ICX-6610). Can I LACP the breakout ports on Stack Unit 1 and Stack Unit 3? And would they then stack? I'll look into doing LACP on the breakout ports.

Also, I read somewhere (on reddit) that someone was using the MCX354A on one of the breakout ports but still getting 40G (although, I think he might have been using a stack-trunk and that 40G was probably from his 1/2/1 or 1/2/6). So maybe that's an option for me, too (to move that connection 1/2/2 or 1/2/7 and then connect the stack to 1/2/6 and 3/2/1).

When I tried to use "stack unit 3" -> stack-trunk 3/2/6 to 3/2/7 it didn't show as a ring topology. It showed as a linear topology with a trunk on the connection between stack unit 2 and stack unit 3 (even though the cables were physically connected from stack unit 3 to stack unit 1).
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,691
1,123
113
Can I LACP the breakout ports on Stack Unit 1 and Stack Unit 3? And would they then stack? I'll look into doing LACP on the breakout ports.
Sure, LAG across the stack is possible (basically the alternative to MLAG / MCT)

Also, I read somewhere (on reddit) that someone was using the MCX354A on one of the breakout ports but still getting 40G (although, I think he might have been using a stack-trunk and that 40G was probably from his 1/2/1 or 1/2/6).
Certainly not. That is not possible. It might have linked, but not with 40G.
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,691
1,123
113
That said, I'm not married to this idea, I can be swayed. My core requirements would be at least 24x 10gb SFPs and 6x 40gb SFPs. Everything else is negotiable.
Very much depends on how important power consumption is for you.
If low power consumption is critical: Mellanox SX1024 (SFP+QSFP) or SX6036 (QSFP only)

If not: Arista 7050SX-72Q (SFP + QSFP) or 7050QX-32(S) (QSFP only, except the 4 SFPs on the -S)

The Aristas are generally nicer than the SX6036 software-wise - the latest available MLNX-OS on the SX6036 is ANCIENT, has some bugs and an ANNOYINGLY SLOW control plane.
 

kapone

Well-Known Member
May 23, 2015
1,285
738
113
The Aristas are generally nicer than the SX6036 software-wise - the latest available MLNX-OS on the SX6036 is ANCIENT, has some bugs and an ANNOYINGLY SLOW control plane.
Aware of that. The low power consumption is appealing though. (I mean, how often do I mess with the control plane? Not much, after initial setup)
 

BoGs

Member
Feb 18, 2019
128
20
18
If not: Arista 7050SX-72Q (SFP + QSFP) or 7050QX-32(S) (QSFP only, except the 4 SFPs on the -S)
QSFP that you can breakout to 4x10G sfp links which is what I used to do. Either dac or get optics which will need MTP fiber cable to 4 LC. Word of warning if you leave the fans on auto the 7050QX min fan is 60% which you can overwrite and I ran it at 35-40% depending on the time of year and even then it runs the asic around 50c. Just so you know. It is an awesome switch.
 

BoGs

Member
Feb 18, 2019
128
20
18
Now, there's of course the Mellanox SX6036 as well...but gotta go through some contortions to get it working right.
Be careful with this one as only some of the ports can be broken out to 4x10 and not all of them (or majority) so you have to make sure you read the documentation, have one of these as well :)
 

jmdomini

New Member
Jan 30, 2024
16
0
1
Thank you both for trying to help. Seems I have bigger problems as I came back to a dead switch. Appears the power supply toasted on my core switch (burning smell). So I've had to put back my old Cisco switch for now. I'll explore my options for replacement after the holidays.
Did an autopsy on the switch (6450-24P) and there's a hole burned in one of the chips on what I'm assuming is the PoE board, and through the other side of the board. I imagine something like this would have blown the fuse on the power supply assuming there is one. I didn't explore any further after finding this catastrophic failure.

Not sure I want to replace it with another Brocade considering I only got about six months out of it. I've never had a Cisco die on me. It's getting harder to use anything but their really ancient stuff for home lab use because of their unfriendly licensing requirements however.
 

Attachments

klui

༺༻
Feb 3, 2019
970
552
93
The QX-32S (S-version) [...] has 4 additional SFP ports.
Unlike the 7060CX-32S and later generations, the 7050X's SFP+ ports are just a convenience breakout. The 7050QX-32S's are tied to port 5. Either the SFP+ ports or QSFP+ port is enabled at any given time.


Can I choose which QSFP+ interface to disable in lieu of the SFP+ ports on the DCS-7050QX-32S?
The leftmost QSFP+ port (Et5/1) must be disabled, as this shares the connections with the 4 SFP+ ports.​
How do I enable/disable the SFP+ ports on 7050QX-32S?
The mode of the first SFP+ and QSFP+ ports is selected as follows:​
Switch(config)#hardware port-group 1 select Et1-4 [SFP+]​
Switch(config)#hardware port-group 1 select Et5/1-4 [QSFP+]​
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,691
1,123
113
Did an autopsy on the switch (6450-24P) and there's a hole burned in one of the chips on what I'm assuming is the PoE board, and through the other side of the board. I imagine something like this would have blown the fuse on the power supply assuming there is one. I didn't explore any further after finding this catastrophic failure.
Classic failure mode on the Brocade / Ruckus PoE switches...
 

kapone

Well-Known Member
May 23, 2015
1,285
738
113
Mellanox SX6036 and various odds and ends (cables, optics etc) ordered. We'll see how it plays out.
 

seatrope

Member
Oct 5, 2018
35
12
8
Maine
www.ychng.com
Looking for some help from people here (esp @kapone you've been amazing in the past!).

I've gotten the standard Opnsense and ICX6610 setup working where all my local traffic between my 3 VLANs are routed internally, anything going to the internet is routed to OpnSense.

However, I just got a Reolink doorbell and very annoyingly, it will not allow local connections unless they appear to come from the same VLAN. The Reolink doorbell is on the IOT VLAN, and client devices connecting to the doorbell are usually on the "STANDARD" or "ADMIN" VLANs.

On the Reolink forums, the solution given was to Source NAT the client IP so that it appears to be coming from the same VLAN as the doorbell.

ICX switches cannot SNAT. So conceptually i think i have to route any traffic from the STANDARD or ADMIN VLANs directed to the Reolink Doorbell IP out to Opnsense for Opnsense to SNAT and send back into the IOT VLAN.

Before I start messing around blindly, I'm guessing i'd have to add an entry to the ICX6610 routing table and also some routing rules/SNAT rules in Opnsense.
Not sure of the exact details.

If someone could guide me on routing table modifications in the ICX6610 to achieve the above that would be great!

Thanks!!
 

seatrope

Member
Oct 5, 2018
35
12
8
Maine
www.ychng.com
Well, i seem to lack a basic understanding of routing tables.

I added a static route for anything destined for the Reolink IP to be sent to Opnsense. Of course that sent anything destined for the Reolink back to Opnsense in an infinite loop, duh.

Is it possible to add a source - destination to the ICX routing tables? or is it just destination only?
 

kapone

Well-Known Member
May 23, 2015
1,285
738
113
lol. No idea where to begin with VRFs…
Well...a few things we can try (without getting into VRFs...)

what "devices" on the STANDARD or ADMIN VLANs are we talking about? Are they VLAN capable natively? (It'd be trivial to do a dual homed device...IF they supported VLANs)
 

seatrope

Member
Oct 5, 2018
35
12
8
Maine
www.ychng.com
Well...a few things we can try (without getting into VRFs...)

what "devices" on the STANDARD or ADMIN VLANs are we talking about? Are they VLAN capable natively? (It'd be trivial to do a dual homed device...IF they supported VLANs)
its ok :) I started reading about VRFs. New thing to learn!

the devices on the other VLANs are iPhones. Can’t dual home :(
 

cyinite

New Member
Jun 28, 2024
8
5
3
Hmm, I guess I'm confused then. It's been a while, but I don't recall having to do any routing to pass VLAN's between switches. I used to have some ancient Cisco switches and I seem to think that I could have say VLAN10 on both switches and assign ports on both. Do switch trunk allowed vlan all on the connection between them and devices connected to ports associated to the same VLAN at both ends could communicate with each other.

Sounds like you are saying it doesn't work that with Brocade switches and I need to somehow tell switch A that VLAN 2 exist on x port on switch B as well. Where as I'm assuming that dual-mode essential works like "switchport trunk allowed vlan all" and I'm expecting VLAN 2 to automagically carry over to switch B because I allowed it to pass on the "trunk" using this command.
It could be the suspect as looking at your running config, it should be fully operational. Heck, I even tried it out with some spare switches I had and it worked as expected but I only own ICX 7xxx switches and my main ICX does the routing.

Thank you both for trying to help. Seems I have bigger problems as I came back to a dead switch. Appears the power supply toasted on my core switch (burning smell). So I've had to put back my old Cisco switch for now. I'll explore my options for replacement after the holidays.
Really unfortunate to hear, you could look into getting a newer Brocade/Ruckus switch (e.g. ICX7150 or ICX7250) if you liked the feature set.