Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

EngChiSTH

Member
Jun 27, 2018
49
18
8
Chicago
So if you look in the 8095 firmware zip, you'll see a new ICX7550 folder - a product that hasn't been announced. I don't think they meant to include it yet, but I'm not sure. The release notes say that no hardware support is added in this GA release, but 8095a will add ICX7550 support. However it's already in there. I had a dig around and found pretty much nothing on the product, except an internal-only confidential marketing report from a Ruckus russian branch, including some great memes -



You can see the full PDF here: https://livemarketolog.ru/upload/iblock/236/23662f70a5408e12516ee20308e73f9b.pdf
Thank you @fohdeesha , interesting read and presentation.
 
  • Like
Reactions: fohdeesha

adman_c

New Member
Feb 14, 2016
4
2
3
Chicago
I thought I had mentioned the FFB0412HN-5T20 in my gigantic fan post of doom a couple pages back, but alas, I did not. I did mention the EFB, but they're very different fans. The FFB's also have a very... distinct harmonic because of the turbulence vanes on that family. So I generally hesitate to give them a blanket recommendation.
Here's the data sheet.
HN-5T20's not on the sheet because '5T20' is the function. Look to HN-R00/F00. 6500RPM, 10.59CFM (pretty high for the RPM,) 0.248inH2O, at 36dBA. However, several sites list the FFB0412 as not recommended for new orders. So Delta may be planning to discontinue it. (Would surprise me though. A LOT of companies use LARGE numbers of FFBs.) The EFB0412VHD-F00 I linked has similar performance but higher RPM, and lower noise due to frame and impeller differences.
As I said: there's a reason Delta is a top pick at every ODM and OEM, yours truly included. FFB0412's a great fan any way you slice it. Just a question of whether or not you like the harmonics. They are very unique.

By the way, I keep neglecting to mention, the reason the KDE1204's resonate is the same reason all MagLevs resonate. I've been positive for years that it's a motor design issue. They were designed for fixed 12V and just tach output. (I literally have piles of KDE1204's because they were the go-to fan for very-long-life fixed voltage applications.) The newer parts have a revised motor that mostly eliminates the harmonic problems when below 12V.
Yes, I should note that I read and very much enjoyed your "fan post of doom." Thanks a lot for that. I even went out and tracked down a couple of the gryphon fans to try out as my next victims. I don't mind the harmonics of the FFB0412s--as I said, they're vastly better than the Sunons they replaced. But I'm still looking for the holy grail here: ok cooling at very-near-silent. One more good thing about the FFB0412's I linked is that they're currently $8 for 2 at amazon, so the cost for determining whether you can tolerate their noise level is pretty low.
 

Spearfoot

Member
Apr 22, 2015
67
26
18
62
I know you can get them on eBay for a reasonable price. I was thinking of listings like this, where the seller is offering an ICX 7150-C12P -- the 1G, not the 10G version -- for $899. That's crazy! You can buy a new one (from Provantage, for example) for $375.

crazy-price-7150-c12p.jpg
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,958
1,783
113
29
fohdeesha.com
yeah the ICX7150 is definitely not going anywhere, in fact from what I understand it's their most popular selling ICX model. According to that russian marketing powerpoint that leaked it looks like they're even introducing a "hardened" version of it for industrial applications sometime before the end of the year
 

tommybackeast

Active Member
Jun 10, 2018
251
82
28
There's no physical licenses for the icx7xxx series, it's just a simple command that fully unlocks them for free. It's all covered in the update/config guide linked in the original post of this thread
fohdeesha : Have a new ICX 7150-C12P incoming :

your software page shows "SW version: 08080e ZIP Updated: 10-29-2019"

A few weeks ago I thought I had read there you were planning to update your suggested software with a newer version -

Thus, question of what version you currently suggest for 7150-c12p : thank you as always
 

dswartz

Active Member
Jul 14, 2011
433
37
28
I'm curious as to the pros and cons of ring vs linear stacking. I've found brocade docs that say ring is more redundant, and linear has one cable connecting each of the switches, but then the very next diagram shows what appears to be a linear stack with *two* cables connecting each pair of switches. Is that not redundant? The docs have a ton of details but don't always explain *why*. Hmmm...
 

ICXGURU

Member
Jun 22, 2020
36
15
8
I'm curious as to the pros and cons of ring vs linear stacking. I've found brocade docs that say ring is more redundant, and linear has one cable connecting each of the switches, but then the very next diagram shows what appears to be a linear stack with *two* cables connecting each pair of switches. Is that not redundant? The docs have a ton of details but don't always explain *why*. Hmmm...
In a two node stack ONLY you want to create a linear stack with a stack trunk because it will create redundancy and double the bandwidth. It is explained in the stacking guide for 2 node stack BUT anything greater than 2 nodes you never want to use linear unless you absolutely have no other choice and even then I would forget stacking and configure separately with VRRPe. The reason is that in the event that the single stack link gets broken the stack will try to avoid a split brain situation (or broken LAGS that cross multiple switches) at all cost so it will actually stop forwarding traffic on the section of the stack that does not have the stack master. This is fine for anything on the master side of the break but obviously very bad for devices on the other side. For the cost of one cable, just do the right thing and finish your ring!
 
  • Like
Reactions: dswartz

ICXGURU

Member
Jun 22, 2020
36
15
8
In a two node stack ONLY you want to create a linear stack with a stack trunk because it will create redundancy and double the bandwidth. It is explained in the stacking guide for 2 node stack BUT anything greater than 2 nodes you never want to use linear unless you absolutely have no other choice and even then I would forget stacking and configure separately with VRRPe. The reason is that in the event that the single stack link gets broken the stack will try to avoid a split brain situation (or broken LAGS that cross multiple switches) at all cost so it will actually stop forwarding traffic on the section of the stack that does not have the stack master. This is fine for anything on the master side of the break but obviously very bad for devices on the other side. For the cost of one cable, just do the right thing and finish your ring!
I should point out that my recommendation for linear stacking for two note stack is only for version 8.0.90 and above... anything before that you should use a ring topology even for two node. See "Support for two-unit stack linear-topology trunks" in the stacking guide.
 
  • Like
Reactions: dswartz

rootwyrm

Member
Mar 25, 2017
40
48
18
www.rootwyrm.com
So, I actually want to pick your collective brains on this one, because it truly has got me stumped.

ICX6450, 3x Juniper EX2200's, single gig link for each EX2200. We already know that the ICX doesn't actually do PVLANs (not the way they actually work.) This is a non-issue though because the EX2200's are carrying specific VLANs due to what's connected to them. Uplink ports on both sides are set to tagged-trunking. The EX2200 either tags internally with native-vlan-id or trunk + 802.1q and pass-through (it gets fairly complicated because of the downstream devices, so I'm simplifying for you all.) When I put a traffic analyzer between the uplink, I see tagged frames flowing both directions, but neither switch acks them. I thought it was MTU chop so I turned on jumbo on the ICX6450 and set the MTU to 1536 on the Juniper side.
Did not help. At all. Did nothing. Also, the ICX6450 doesn't seem to be MTU probing properly either - sets to 10200 even though the Juniper responds to probes as 1536. And remember, not stacking here, just passing VLANs between switches. Port state is good on both sides, so I'm just completely stumped here as to why they aren't handling frames or MTU correctly. (Also, WTF crack was Brocade smoking when they made it an all-or-nothing 1500 or 10200 with no functioning PTMUD? That's just inexcusably stupid.)
 

dswartz

Active Member
Jul 14, 2011
433
37
28
Thanks! Sorry for being unclear, I was always planning on using two cables. I just didn't understand what the difference was between ring and linear, given that the one example showed a 3-node linear stack with two cables between each switch.
 

ICXGURU

Member
Jun 22, 2020
36
15
8
So, I actually want to pick your collective brains on this one, because it truly has got me stumped.

ICX6450, 3x Juniper EX2200's, single gig link for each EX2200. We already know that the ICX doesn't actually do PVLANs (not the way they actually work.) This is a non-issue though because the EX2200's are carrying specific VLANs due to what's connected to them. Uplink ports on both sides are set to tagged-trunking. The EX2200 either tags internally with native-vlan-id or trunk + 802.1q and pass-through (it gets fairly complicated because of the downstream devices, so I'm simplifying for you all.) When I put a traffic analyzer between the uplink, I see tagged frames flowing both directions, but neither switch acks them. I thought it was MTU chop so I turned on jumbo on the ICX6450 and set the MTU to 1536 on the Juniper side.
Did not help. At all. Did nothing. Also, the ICX6450 doesn't seem to be MTU probing properly either - sets to 10200 even though the Juniper responds to probes as 1536. And remember, not stacking here, just passing VLANs between switches. Port state is good on both sides, so I'm just completely stumped here as to why they aren't handling frames or MTU correctly. (Also, WTF crack was Brocade smoking when they made it an all-or-nothing 1500 or 10200 with no functioning PTMUD? That's just inexcusably stupid.)
maybe I missed this discussion... what's wrong with PVLANs on the ICX?
 

rootwyrm

Member
Mar 25, 2017
40
48
18
www.rootwyrm.com
maybe I missed this discussion... what's wrong with PVLANs on the ICX?
The ICX simply put, does not do PVLANs by the actual definition. Words have meanings no matter how much Brocade tried to redefine them.
PVLANs are, and have always been, a VLAN within a VLAN. Specifically to get around the limit of 4095 VLANs. i.e. on an actual PVLAN capable switch, VLAN 2000 is actually a PVLAN that encapsulates VLAN 100-110 so as to not interfere with real VLAN 100. Isolated VLANs are not PVLANs!! THEY ARE NON-SWITCHING VLANS! NOT THE SAME.
So the actual architecture looks like this:
Code:
        +------- Switching/Routing Plane <--> VLAN100-VLAN110
VLAN100 VLAN2000(VLAN100-110)
+-------+------- Switching Plane
VLAN100 VLAN2000(VLAN100-110)
        +------- Switching/Routing Plane <--> VLAN100-VLAN110
Yes it's hard to show in ASCII. So have a much more useful image from Juniper.


PVLANs are, by definition, VLAN trunks that encapsulate VLANs. ICX straight up cannot do this. (This should surprise nobody who has looked at the encapsulation or isolation capabilities of any Brocade product. Especially DCX.) So my ONLY choice is to use tagging. In futile hopes of reducing migraines (and avoid loops) the Junipers aren't doing any ISLs or vchassis either.
And the devices behind the Junipers literally cannot connect to any ICX due to known defects/lack-of-support with the Ethernet silicon. Juniper is a "divorced" architecture. Juniper is individual Ethernet MAC+PHY to build a fabric, and the 7 series is a fully integrated fabric+Ethernet package using BCM Ethernet. And even getting these devices to talk to the Juniper was a whole thing. It's straight up impossible with any ICX. You'll find this with a whole lot of older devices that they will not talk to stuff like the ICX 7 series and only spotty with the 6 series, particularly 100bT devices or anything that needs more rational MTU/MSS.
 

ICXGURU

Member
Jun 22, 2020
36
15
8
uh yeah....There are two definitions for PVLANs. You seem to think that PVLAN can only stand for a provider vlan for 802.1ad (Q-in-Q tunnels) which Brocade does support even on the 6450 if you check the documentation but the most common use for the term PVLAN refers to private vlans which Brocade, Juniper and Cisco, and well pretty much everyone uses as an acronym.
 
  • Like
Reactions: Jason Antes

rootwyrm

Member
Mar 25, 2017
40
48
18
www.rootwyrm.com
uh yeah....There are two definitions for PVLANs. You seem to think that PVLAN can only stand for a provider vlan for 802.1ad (Q-in-Q tunnels) which Brocade does support even on the 6450 if you check the documentation but the most common use for the term PVLAN refers to private vlans which Brocade, Juniper and Cisco, and well pretty much everyone uses as an acronym.
Except I've already confirmed that the 6450 does not behave at all correctly or sanely when you introduce QinQ at all, which is exacerbated by the completely non-functional PMTUD and absolutely insane expectation that anybody does 10200 jumbo frames. (NOPE.)
Either way that's getting away from the fact that it's not passing basic 802.1q VLANs between switches correctly. Which is the problem.
 

ICXGURU

Member
Jun 22, 2020
36
15
8
Except I've already confirmed that the 6450 does not behave at all correctly or sanely when you introduce QinQ at all, which is exacerbated by the completely non-functional PMTUD and absolutely insane expectation that anybody does 10200 jumbo frames. (NOPE.)
Either way that's getting away from the fact that it's not passing basic 802.1q VLANs between switches correctly. Which is the problem.
strange, I have deployed Q-in-Q between Brocade and Cisco many times and it works fine. Perhaps it is the Juniper not following the standard?

anyway, back to the topic. I don't know for certain if the 6450 supports path MTU discovery. I can say that the 6430 does not and every other Foundry, Brocade, Ruckus switch does for sure including the nearly 20 year old FCX, and SuperX. It is remotely possible that the ASIC did not but what you can do is set the per port MTU. The default without jumbo is going to be 1500 bytes and with jumbo is 9216 bytes. Just change the interface either on the physical interface or VE depending if it has one or not.

interface ethernet 1/2/1
ip mtu 1536

You may also want to try setting the mtu-exceed to forward rather than drop on the interface.
mtu-exceed forward

A standard for jumbo MTU is either 9000 or 9216 in almost all datacenter storage environments. Which one varies by storage and server vendor. I know 10200 is higher than most vendors hardware is capable of but the default on the interface is 9216.
 
  • Like
Reactions: Jason Antes

safado

New Member
Aug 21, 2020
15
0
1
Maybe a stupid question but I’m curious and will ask anyway for the sake of learning.

I initially tried some FS transceivers for a 10gBase-T to my workstation over a ~30-50m CAT5e run and it didn’t work. Before deciding to order a spool of Cat6a and add a run I saw they had a 50m (which i ordered) and then saw a 80m model. Took a shot at it working before dropping 250 on a spool and to my surprise it works perfectly. Getting 600mbps transfers to my NAS.

My question is what makes the 50m vs 80m transceiver work? Better build quality or does the 80m utilize more power? They look identical to me—just curious how this works? Thank you for the knowledge!
 

infoMatt

Active Member
Apr 16, 2019
166
65
28
When I put a traffic analyzer between the uplink, I see tagged frames flowing both directions, but neither switch acks them.
What do you mean with this sentence? The L2 frames aren't acked back to the sender... or do you mean that although you see the packets there isn't actually any connectivity between devices connected to the two switches?