Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
yeah the ICX7150 is definitely not going anywhere, in fact from what I understand it's their most popular selling ICX model. According to that russian marketing powerpoint that leaked it looks like they're even introducing a "hardened" version of it for industrial applications sometime before the end of the year
 
  • Like
Reactions: tommybackeast

tommybackeast

Active Member
Jun 10, 2018
286
105
43
There's no physical licenses for the icx7xxx series, it's just a simple command that fully unlocks them for free. It's all covered in the update/config guide linked in the original post of this thread
fohdeesha : Have a new ICX 7150-C12P incoming :

your software page shows "SW version: 08080e ZIP Updated: 10-29-2019"

A few weeks ago I thought I had read there you were planning to update your suggested software with a newer version -

Thus, question of what version you currently suggest for 7150-c12p : thank you as always
 

dswartz

Active Member
Jul 14, 2011
610
79
28
I'm curious as to the pros and cons of ring vs linear stacking. I've found brocade docs that say ring is more redundant, and linear has one cable connecting each of the switches, but then the very next diagram shows what appears to be a linear stack with *two* cables connecting each pair of switches. Is that not redundant? The docs have a ton of details but don't always explain *why*. Hmmm...
 

ICXGURU

Member
Jun 22, 2020
37
16
8
I'm curious as to the pros and cons of ring vs linear stacking. I've found brocade docs that say ring is more redundant, and linear has one cable connecting each of the switches, but then the very next diagram shows what appears to be a linear stack with *two* cables connecting each pair of switches. Is that not redundant? The docs have a ton of details but don't always explain *why*. Hmmm...
In a two node stack ONLY you want to create a linear stack with a stack trunk because it will create redundancy and double the bandwidth. It is explained in the stacking guide for 2 node stack BUT anything greater than 2 nodes you never want to use linear unless you absolutely have no other choice and even then I would forget stacking and configure separately with VRRPe. The reason is that in the event that the single stack link gets broken the stack will try to avoid a split brain situation (or broken LAGS that cross multiple switches) at all cost so it will actually stop forwarding traffic on the section of the stack that does not have the stack master. This is fine for anything on the master side of the break but obviously very bad for devices on the other side. For the cost of one cable, just do the right thing and finish your ring!
 
  • Like
Reactions: dswartz

ICXGURU

Member
Jun 22, 2020
37
16
8
In a two node stack ONLY you want to create a linear stack with a stack trunk because it will create redundancy and double the bandwidth. It is explained in the stacking guide for 2 node stack BUT anything greater than 2 nodes you never want to use linear unless you absolutely have no other choice and even then I would forget stacking and configure separately with VRRPe. The reason is that in the event that the single stack link gets broken the stack will try to avoid a split brain situation (or broken LAGS that cross multiple switches) at all cost so it will actually stop forwarding traffic on the section of the stack that does not have the stack master. This is fine for anything on the master side of the break but obviously very bad for devices on the other side. For the cost of one cable, just do the right thing and finish your ring!
I should point out that my recommendation for linear stacking for two note stack is only for version 8.0.90 and above... anything before that you should use a ring topology even for two node. See "Support for two-unit stack linear-topology trunks" in the stacking guide.
 
  • Like
Reactions: dswartz

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
So, I actually want to pick your collective brains on this one, because it truly has got me stumped.

ICX6450, 3x Juniper EX2200's, single gig link for each EX2200. We already know that the ICX doesn't actually do PVLANs (not the way they actually work.) This is a non-issue though because the EX2200's are carrying specific VLANs due to what's connected to them. Uplink ports on both sides are set to tagged-trunking. The EX2200 either tags internally with native-vlan-id or trunk + 802.1q and pass-through (it gets fairly complicated because of the downstream devices, so I'm simplifying for you all.) When I put a traffic analyzer between the uplink, I see tagged frames flowing both directions, but neither switch acks them. I thought it was MTU chop so I turned on jumbo on the ICX6450 and set the MTU to 1536 on the Juniper side.
Did not help. At all. Did nothing. Also, the ICX6450 doesn't seem to be MTU probing properly either - sets to 10200 even though the Juniper responds to probes as 1536. And remember, not stacking here, just passing VLANs between switches. Port state is good on both sides, so I'm just completely stumped here as to why they aren't handling frames or MTU correctly. (Also, WTF crack was Brocade smoking when they made it an all-or-nothing 1500 or 10200 with no functioning PTMUD? That's just inexcusably stupid.)
 

dswartz

Active Member
Jul 14, 2011
610
79
28
Thanks! Sorry for being unclear, I was always planning on using two cables. I just didn't understand what the difference was between ring and linear, given that the one example showed a 3-node linear stack with two cables between each switch.
 

ICXGURU

Member
Jun 22, 2020
37
16
8
So, I actually want to pick your collective brains on this one, because it truly has got me stumped.

ICX6450, 3x Juniper EX2200's, single gig link for each EX2200. We already know that the ICX doesn't actually do PVLANs (not the way they actually work.) This is a non-issue though because the EX2200's are carrying specific VLANs due to what's connected to them. Uplink ports on both sides are set to tagged-trunking. The EX2200 either tags internally with native-vlan-id or trunk + 802.1q and pass-through (it gets fairly complicated because of the downstream devices, so I'm simplifying for you all.) When I put a traffic analyzer between the uplink, I see tagged frames flowing both directions, but neither switch acks them. I thought it was MTU chop so I turned on jumbo on the ICX6450 and set the MTU to 1536 on the Juniper side.
Did not help. At all. Did nothing. Also, the ICX6450 doesn't seem to be MTU probing properly either - sets to 10200 even though the Juniper responds to probes as 1536. And remember, not stacking here, just passing VLANs between switches. Port state is good on both sides, so I'm just completely stumped here as to why they aren't handling frames or MTU correctly. (Also, WTF crack was Brocade smoking when they made it an all-or-nothing 1500 or 10200 with no functioning PTMUD? That's just inexcusably stupid.)
maybe I missed this discussion... what's wrong with PVLANs on the ICX?
 

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
maybe I missed this discussion... what's wrong with PVLANs on the ICX?
The ICX simply put, does not do PVLANs by the actual definition. Words have meanings no matter how much Brocade tried to redefine them.
PVLANs are, and have always been, a VLAN within a VLAN. Specifically to get around the limit of 4095 VLANs. i.e. on an actual PVLAN capable switch, VLAN 2000 is actually a PVLAN that encapsulates VLAN 100-110 so as to not interfere with real VLAN 100. Isolated VLANs are not PVLANs!! THEY ARE NON-SWITCHING VLANS! NOT THE SAME.
So the actual architecture looks like this:
Code:
        +------- Switching/Routing Plane <--> VLAN100-VLAN110
VLAN100 VLAN2000(VLAN100-110)
+-------+------- Switching Plane
VLAN100 VLAN2000(VLAN100-110)
        +------- Switching/Routing Plane <--> VLAN100-VLAN110
Yes it's hard to show in ASCII. So have a much more useful image from Juniper.


PVLANs are, by definition, VLAN trunks that encapsulate VLANs. ICX straight up cannot do this. (This should surprise nobody who has looked at the encapsulation or isolation capabilities of any Brocade product. Especially DCX.) So my ONLY choice is to use tagging. In futile hopes of reducing migraines (and avoid loops) the Junipers aren't doing any ISLs or vchassis either.
And the devices behind the Junipers literally cannot connect to any ICX due to known defects/lack-of-support with the Ethernet silicon. Juniper is a "divorced" architecture. Juniper is individual Ethernet MAC+PHY to build a fabric, and the 7 series is a fully integrated fabric+Ethernet package using BCM Ethernet. And even getting these devices to talk to the Juniper was a whole thing. It's straight up impossible with any ICX. You'll find this with a whole lot of older devices that they will not talk to stuff like the ICX 7 series and only spotty with the 6 series, particularly 100bT devices or anything that needs more rational MTU/MSS.
 

ICXGURU

Member
Jun 22, 2020
37
16
8
uh yeah....There are two definitions for PVLANs. You seem to think that PVLAN can only stand for a provider vlan for 802.1ad (Q-in-Q tunnels) which Brocade does support even on the 6450 if you check the documentation but the most common use for the term PVLAN refers to private vlans which Brocade, Juniper and Cisco, and well pretty much everyone uses as an acronym.
 
  • Like
Reactions: Jason Antes

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
uh yeah....There are two definitions for PVLANs. You seem to think that PVLAN can only stand for a provider vlan for 802.1ad (Q-in-Q tunnels) which Brocade does support even on the 6450 if you check the documentation but the most common use for the term PVLAN refers to private vlans which Brocade, Juniper and Cisco, and well pretty much everyone uses as an acronym.
Except I've already confirmed that the 6450 does not behave at all correctly or sanely when you introduce QinQ at all, which is exacerbated by the completely non-functional PMTUD and absolutely insane expectation that anybody does 10200 jumbo frames. (NOPE.)
Either way that's getting away from the fact that it's not passing basic 802.1q VLANs between switches correctly. Which is the problem.
 

ICXGURU

Member
Jun 22, 2020
37
16
8
Except I've already confirmed that the 6450 does not behave at all correctly or sanely when you introduce QinQ at all, which is exacerbated by the completely non-functional PMTUD and absolutely insane expectation that anybody does 10200 jumbo frames. (NOPE.)
Either way that's getting away from the fact that it's not passing basic 802.1q VLANs between switches correctly. Which is the problem.
strange, I have deployed Q-in-Q between Brocade and Cisco many times and it works fine. Perhaps it is the Juniper not following the standard?

anyway, back to the topic. I don't know for certain if the 6450 supports path MTU discovery. I can say that the 6430 does not and every other Foundry, Brocade, Ruckus switch does for sure including the nearly 20 year old FCX, and SuperX. It is remotely possible that the ASIC did not but what you can do is set the per port MTU. The default without jumbo is going to be 1500 bytes and with jumbo is 9216 bytes. Just change the interface either on the physical interface or VE depending if it has one or not.

interface ethernet 1/2/1
ip mtu 1536

You may also want to try setting the mtu-exceed to forward rather than drop on the interface.
mtu-exceed forward

A standard for jumbo MTU is either 9000 or 9216 in almost all datacenter storage environments. Which one varies by storage and server vendor. I know 10200 is higher than most vendors hardware is capable of but the default on the interface is 9216.
 
  • Like
Reactions: Jason Antes

safado

Member
Aug 21, 2020
44
6
8
Maybe a stupid question but I’m curious and will ask anyway for the sake of learning.

I initially tried some FS transceivers for a 10gBase-T to my workstation over a ~30-50m CAT5e run and it didn’t work. Before deciding to order a spool of Cat6a and add a run I saw they had a 50m (which i ordered) and then saw a 80m model. Took a shot at it working before dropping 250 on a spool and to my surprise it works perfectly. Getting 600mbps transfers to my NAS.

My question is what makes the 50m vs 80m transceiver work? Better build quality or does the 80m utilize more power? They look identical to me—just curious how this works? Thank you for the knowledge!
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
When I put a traffic analyzer between the uplink, I see tagged frames flowing both directions, but neither switch acks them.
What do you mean with this sentence? The L2 frames aren't acked back to the sender... or do you mean that although you see the packets there isn't actually any connectivity between devices connected to the two switches?
 

virulent

Member
Jul 3, 2019
28
39
13
Finally caved on an ICX6450-24p.. amazing piece of kit that will replace my mixed ubiquiti + CRS305 setup. Love ubiquiti but hate the direction, so I'm glad to replace everything with this.

There's 1 left if anyone else is looking for a 24poe Brocade ICX ICX6450 24P Gigabit Switch with SFP+, rackears, | eBay

They went down to US$150.. going by sold items, it's pretty on par for sold 48poes but still way less than current listings -- a decent deal. GSP shipping to Canada was actually pretty reasonable (compared to some others that are $100+ lol) so I caved.
 

ICXGURU

Member
Jun 22, 2020
37
16
8
Maybe a stupid question but I’m curious and will ask anyway for the sake of learning.

I initially tried some FS transceivers for a 10gBase-T to my workstation over a ~30-50m CAT5e run and it didn’t work. Before deciding to order a spool of Cat6a and add a run I saw they had a 50m (which i ordered) and then saw a 80m model. Took a shot at it working before dropping 250 on a spool and to my surprise it works perfectly. Getting 600mbps transfers to my NAS.

My question is what makes the 50m vs 80m transceiver work? Better build quality or does the 80m utilize more power? They look identical to me—just curious how this works? Thank you for the knowledge!
It's a bit of a mystery what the difference is but since FS says the 80m will not work on Cat5e and needs Cat6a my guess would be it is just receive sensitivity and maybe more efficient chip set with lower power consumption to output. The extra shielding on the Cat6a would provide a cleaner signal over distance with the same amount of power. I have read multiple people say that they are compatible with each other as long as you do not exceed the shortest length so you can use a 30m at one end and 50m, 80m or 100m at the other as long as your cable does not exceed 30m. That tells me it is the same frequency and pairs.
 
  • Like
Reactions: safado

richrichgreen

New Member
Sep 18, 2020
3
0
1
I have a question, I need 3(or 4) 40Gbe ports. Would stacking 2 ICX6610 result in 4 x 40gbe ports or just 2, using the other 2 40gbe ports for stacking? To be more specific, can you use the breakout 40Gbe ports for stacking and get a more than 10Gbps stream between switches
 
Last edited: