Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

kh78

New Member
Mar 31, 2020
27
6
3
Jumbo Frames. Don't run away...

Anyone else see SSH issues after globally enabling them?
I'm using the MGMT interface on the back to SSH to the ICX 6610. The interface MTU is 1500:

sh int man 1 | i MTU
Internet address is xxx.xxx.xxx.xxx/24, MTU 1500 bytes, encapsulation ethernet


If I do the following, it crashes the SSH session. I don't recall that this happened prior to globally enabling jumbo frames.

conf t
int e x/x/x
ip ?


It crashes the session before actually displaying the question mark.
Other commands with lots of output (but not show commands, they're paginated or something via less or similar) also crash the session. For what it's worth, the NIC on this PC is configured for an MTU of 9000. The terminal is Token2Shell on Windows 10, also crashes XShell. Don't recall having this problem with some other access switches (SG-300's) with jumbo frames enabled, or on the ICX 6610 prior to enabling jumbo frames.

It's like the switch is originating its own larger MTU in response to the command '?", but is then dropping that frame on egress from it's MGMT interface due to the 1500 MTU (which can't be changed anyway) rather than fragmenting the packet...and somehow killing SSH in the process?

I do have an inband mgmt vlan and ve with 'ip mtu 1500' avaliable as well, might try accessing via that and see if it does the same...
******
EDIT:
So, global jumbo frames don't break the in-band SSH management. At least, not as long as you have the ip mtu 1500 set in the ve for the vlan as I do. The physical mgmt port remains broken. I can't be bothered rolling back jumbo globally and then reconfiguring all the vlans and int mtu's again just to confirm as this is an in use switch, but if someone with the same switch/software could check if they have a spare test switch, it might save someone a headache in future.
******

Switch Software:


sh ver

Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
(10545591 bytes) from Primary FCXR08030u.bin
SW: Version 08.0.30uT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #:
License: ICX6610_ADV_ROUTER_SOFT_PACKAGE (LID: )
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,958
1,783
113
29
fohdeesha.com
Guess I asked for that lol..how many pages when you chop out all of the incredibly specific troubleshooting/scenario chaff?
at the bottom of the first post is a list of "helpful posts", read those. specifically the one that links to videos / terry henry's channel, those short videos will be the most helpful
 
  • Like
Reactions: zunder1990

kh78

New Member
Mar 31, 2020
27
6
3
jumbo frames in 2020 = ban
Yep, I'm coming around to that.

Decided I'll just run FDR IB on the SAN interfaces, go to 1500 mtu eth for the trunk to the Brocade, and run 1500 everywhere from there on out in the access layer. Proving to be a nightmare with the jumbo's on the ICX. Liking the ICX otherwise though, really nice CLI arrangement and methods.


at the bottom of the first post is a list of "helpful posts", read those. specifically the one that links to videos / terry henry's channel, those short videos will be the most helpful
Second Terry Henry's channel, great stuff.
Also: Index of /fastiron/08.0.30
 
  • Like
Reactions: fohdeesha

Epsilonson

New Member
Jul 29, 2020
3
0
1
at the bottom of the first post is a list of "helpful posts", read those. specifically the one that links to videos / terry henry's channel, those short videos will be the most helpful
Ah thanks, that's more what I was looking for. I had read a couple of the other ones but missed the videos.


Beautiful, that will save me tons of time. You are awesome!
 

infoMatt

Active Member
Apr 16, 2019
166
65
28
jumbo frames in 2020 = ban
Yes... and no...
There are some cases where jumbo frames saves the day, say for example on a SDN where you have an overlay made of VXLANs... to enable MTU 1500 inside the virtual LANs, you'll need jumbo enabled on the physical transport network, otherwise the VXLAN overhead will "eat into" the maximum usable MTU of the "payload" traffic.
Same if you use any sort of tunneling/GRE/IPSec and what not, and need to carry 1500byte packets.

Soooo.. no, this time around I'll disagree with your opinion. I agree btw that in a homelab their use is rarely justifiable, but in a broader scenario, they have their places.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,958
1,783
113
29
fohdeesha.com
Oh sure, increased MTU for tunneling and encapsulation is a whole other ballgame ( and of course unavoidable). When I hear and refer to "jumbo frames" I'm talking about just blindly cranking everything to 9000 in hopes to speed up transfers and crap like we used to have to do in 2010
 
  • Like
Reactions: infoMatt

kh78

New Member
Mar 31, 2020
27
6
3
Just trim off the version number for more recent manuals:

Didn't link to it though as my understanding was the last version suitable was 8.0.30(u, from memory). Can newer versions be run on the 6610 and 6650?
 

kh78

New Member
Mar 31, 2020
27
6
3
Soooo.. no, this time around I'll disagree with your opinion. I agree btw that in a homelab their use is rarely justifiable, but in a broader scenario, they have their places.
In my case, nearly all of that happens in virtual switching within the servers. Where it doesn't, they're on the IB switch with the SAN and can talk to each other over that. The only reason for jumbo on Eth was to speed up the access layer. Not worth the pain in this case, would rather have reliable OOB management than 5 odd percent or whatever improvement that one might reap in terms of goodput.

As noted, inband worked after setting the vlan ip mtu, if that takes your fancy, but I'd rather reduce the chances of remote throat cutting.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,958
1,783
113
29
fohdeesha.com
Just trim off the version number for more recent manuals:

Didn't link to it though as my understanding was the last version suitable was 8.0.30(u, from memory). Can newer versions be run on the 6610 and 6650?
Latest 6 series will run is 8030 indeed. That link has the first revision of the 8030 manuals up to the b patch release. The latest manuals direct from ruckuses site are 8030h if I remember right, and include details about new commands etc added between the initial 8030 release and now. Those are what's included in the firmware zip
 
  • Like
Reactions: kh78

dragonian

New Member
Jan 3, 2020
10
0
1
I'm trying to figure out the best or easiest way to fix my inter-vlan routing issues.

I currently an using a OPNsense firewall/ router (Protectli) for the router on the stick paradigm connected via 2x 1GbE LAG to ICX6450. I'm not trying to do anything crazy for VLANs, just LAN, GUEST, MGMT, and CAMERA.
I am seeing an issue with WAN timeouts when the router is forced to route from CAMERA to LAN (for storage).

I'd like to keep opnsense for most DHCP, firewall duties, IPV6, multicast, etc. I'm sure that a lot of this is due to the nice GUI, and visualizations. Maybe this is wrong but i have fear that the 5 year old L3 routing code is not always going to be sufficient.

I've looked a lot of posts in this thread with similar topics, but haven't seen a "good solution" [in my probably flawed opinion]. Re: here or here , etc.

I had hoped that the LAG would give another path for WAN packets for streaming music & skype connections to not be interfered when the CAMERA copy is taking place, but that doesn't seem to be the case. [Un]fortunately my NAS can sustain ~350 MBps writes, so bi directional will kill the 1 GbE link.
10 GbE is not an option for this firewall box at the moment.
I don't think this issue is CPU bound on the opnsense box.. It gets up to 40-50%.. unless it's a single core issue.

Is there some LAG configuration that I could use to make this connection better?
Would some QoS PCP values make anything better?
I'd probably even consider limiting the bandwidth coming out of the CAMERA vlan/intfc. Is there a good way to do that?

Thanks!
I don't love it.. but at least for the interim, I put a rate limit on the interface. This solves the critical need of not destroying skype/zoom calls at inappropriate times.

Code:
#int e 1/1/20
#rate-limit in fixed 500000
Rate Limiting on Port 1/1/20 - Config: 500000 kbps, Actual: 500000 kbps.
 

kh78

New Member
Mar 31, 2020
27
6
3
Code:
#int e 1/1/20
#rate-limit in fixed 500000
Rate Limiting on Port 1/1/20 - Config: 500000 kbps, Actual: 500000 kbps.
I haven't played with any QoS related stuff on the Brocade as yet, but generally you should be able to tag traffic on ingress in to a switch interface from the source. You then shape (or at least, police) the tagged traffic when it hits another interface.

If you are lucky, the Brocade would support shaping/policing on egress traffic as it exits the switch on the LAG to the FW. If not, you have to rely on the shaping/policing (based on the tags applied to the traffic on ingress from the source to the switch) of the ingress interfaces of the next device (i.e. your firewall in this case). Most firewalls support some form of QoS, including pfsense (though never tried doing it on a LAG, just single interfaces). OPNsense is a pfSense fork, so it should support something similar. Whether you are doing it at layer 2 or layer 3 wouldn't really matter since your not controlling a particular type of traffic, just anything and everything source from a specific interface (i.e. camera networks switch interface).

pfSense claim it can handle up to a few Gbps when routing on average PC hardware. I've never seen that much from it personally, but last time I tested it (5 years ago), it was with hardware that was already about 4 years old at that time (~600mhz firewall box). It crapped out cpu bound at about 750mbps. If you need more than that, maybe a dedicated routing distro is the way to go, or else look at Netgate's TNSR.

If you are super keen, you could try rolling your own with DPDK and Vector Packet Routing etc, which is basically what is providing the performance enhancements to TNSR over pfSense. Personally, I'd just pay for TNSR rather than attempt to DIY it.

An alternative would be to look at doing in virtualised switching, OpenVSwitch comes to mind as the simplest to setup, probably with sufficient capability to do what you want with minimal overheads.
 
Last edited:

shremi

New Member
Jun 29, 2020
1
0
1
Hey guys newbie here.

Thanks to this amazing thread and everyone here i saved almost a grand and instead of buying the Ubiquity 48 Pro Switch i went with the 6450-48p

I still have a long time before the switch comes here and i am starting to think of how i am going to quiet this baby.

Has anyone here attempted a 12 o 14 inch fan mod on top of the switch ?

I know i have a ton of them from my pcs good quality fans used also for watercooling and static pressure is king i still plan on ordering some MB40201VX in case my plan doesn't go as well.

Im curious to know your thoughts on this
 

Spearfoot

Member
Apr 22, 2015
67
26
18
62
Jumbo Frames. Don't run away...

Anyone else see SSH issues after globally enabling them?
I'm using the MGMT interface on the back to SSH to the ICX 6610. The interface MTU is 1500:

sh int man 1 | i MTU
Internet address is xxx.xxx.xxx.xxx/24, MTU 1500 bytes, encapsulation ethernet


If I do the following, it crashes the SSH session. I don't recall that this happened prior to globally enabling jumbo frames.

conf t
int e x/x/x
ip ?


It crashes the session before actually displaying the question mark.
Other commands with lots of output (but not show commands, they're paginated or something via less or similar) also crash the session. For what it's worth, the NIC on this PC is configured for an MTU of 9000. The terminal is Token2Shell on Windows 10, also crashes XShell. Don't recall having this problem with some other access switches (SG-300's) with jumbo frames enabled, or on the ICX 6610 prior to enabling jumbo frames.

It's like the switch is originating its own larger MTU in response to the command '?", but is then dropping that frame on egress from it's MGMT interface due to the 1500 MTU (which can't be changed anyway) rather than fragmenting the packet...and somehow killing SSH in the process?

I do have an inband mgmt vlan and ve with 'ip mtu 1500' avaliable as well, might try accessing via that and see if it does the same...
******
EDIT:
So, global jumbo frames don't break the in-band SSH management. At least, not as long as you have the ip mtu 1500 set in the ve for the vlan as I do. The physical mgmt port remains broken. I can't be bothered rolling back jumbo globally and then reconfiguring all the vlans and int mtu's again just to confirm as this is an in use switch, but if someone with the same switch/software could check if they have a spare test switch, it might save someone a headache in future.
******

Switch Software:


sh ver

Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
(10545591 bytes) from Primary FCXR08030u.bin
SW: Version 08.0.30uT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #:
License: ICX6610_ADV_ROUTER_SOFT_PACKAGE (LID: )
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
Have you tried mtu-exceed forward in your configuration? I believe this changes the switch's default behavior of dropping packets larger than a port's MTU, so that it will instead fragment and pass them along.
 

nev_neo

Member
Jul 31, 2013
141
23
18
Personally I feel that whoever even suggests or asks about implementation of Jumbo Frames in any way, needs to be violently and firmly shutdown.
UGH, i'm still putting out weird Vmware distribution switch warnings months have removing Jumbo frames from the systems.
I wasn't able to see any advantage from going with MTU 9000 and I rue the day that I decided to try it out in our dev environment.
 
  • Like
Reactions: fohdeesha