Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Umm...yeah...about 204 pages worth.
Grab a cold one.Guess I asked for that lol..how many pages when you chop out all of the incredibly specific troubleshooting/scenario chaff?Umm...yeah...about 204 pages worth.
Jumbo Frames. Don't run away...
Anyone else see SSH issues after globally enabling them?
I'm using the MGMT interface on the back to SSH to the ICX 6610. The interface MTU is 1500:
If I do the following, it crashes the SSH session. I don't recall that this happened prior to globally enabling jumbo frames.
It crashes the session before actually displaying the question mark.
Other commands with lots of output (but not show commands, they're paginated or something via less or similar) also crash the session. For what it's worth, the NIC on this PC is configured for an MTU of 9000. The terminal is Token2Shell on Windows 10, also crashes XShell. Don't recall having this problem with some other access switches (SG-300's) with jumbo frames enabled, or on the ICX 6610 prior to enabling jumbo frames.
It's like the switch is originating its own larger MTU in response to the command '?", but is then dropping that frame on egress from it's MGMT interface due to the 1500 MTU (which can't be changed anyway) rather than fragmenting the packet...and somehow killing SSH in the process?
I do have an inband mgmt vlan and ve with 'ip mtu 1500' avaliable as well, might try accessing via that and see if it does the same...
******
EDIT:
So, global jumbo frames don't break the in-band SSH management. At least, not as long as you have the ip mtu 1500 set in the ve for the vlan as I do. The physical mgmt port remains broken. I can't be bothered rolling back jumbo globally and then reconfiguring all the vlans and int mtu's again just to confirm as this is an in use switch, but if someone with the same switch/software could check if they have a spare test switch, it might save someone a headache in future.
******
Switch Software:
Anyone else see SSH issues after globally enabling them?
I'm using the MGMT interface on the back to SSH to the ICX 6610. The interface MTU is 1500:
sh int man 1 | i MTU
Internet address is xxx.xxx.xxx.xxx/24, MTU 1500 bytes, encapsulation ethernet
If I do the following, it crashes the SSH session. I don't recall that this happened prior to globally enabling jumbo frames.
conf t
int e x/x/x
ip ?It crashes the session before actually displaying the question mark.
Other commands with lots of output (but not show commands, they're paginated or something via less or similar) also crash the session. For what it's worth, the NIC on this PC is configured for an MTU of 9000. The terminal is Token2Shell on Windows 10, also crashes XShell. Don't recall having this problem with some other access switches (SG-300's) with jumbo frames enabled, or on the ICX 6610 prior to enabling jumbo frames.
It's like the switch is originating its own larger MTU in response to the command '?", but is then dropping that frame on egress from it's MGMT interface due to the 1500 MTU (which can't be changed anyway) rather than fragmenting the packet...and somehow killing SSH in the process?
I do have an inband mgmt vlan and ve with 'ip mtu 1500' avaliable as well, might try accessing via that and see if it does the same...
******
EDIT:
So, global jumbo frames don't break the in-band SSH management. At least, not as long as you have the ip mtu 1500 set in the ve for the vlan as I do. The physical mgmt port remains broken. I can't be bothered rolling back jumbo globally and then reconfiguring all the vlans and int mtu's again just to confirm as this is an in use switch, but if someone with the same switch/software could check if they have a spare test switch, it might save someone a headache in future.
******
Switch Software:
sh ver
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
(10545591 bytes) from Primary FCXR08030u.bin
SW: Version 08.0.30uT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #:
License: ICX6610_ADV_ROUTER_SOFT_PACKAGE (LID: )
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
Last edited:
at the bottom of the first post is a list of "helpful posts", read those. specifically the one that links to videos / terry henry's channel, those short videos will be the most helpful
Yep, I'm coming around to that.
Decided I'll just run FDR IB on the SAN interfaces, go to 1500 mtu eth for the trunk to the Brocade, and run 1500 everywhere from there on out in the access layer. Proving to be a nightmare with the jumbo's on the ICX. Liking the ICX otherwise though, really nice CLI arrangement and methods.
Second Terry Henry's channel, great stuff.
Also: Index of /fastiron/08.0.30
Ah thanks, that's more what I was looking for. I had read a couple of the other ones but missed the videos.
Beautiful, that will save me tons of time. You are awesome!
and don't forget the firmware zip you should have downloaded from the update/config guide, comes with a "manuals" folder with newer versions of everything at that link and more. A lot of people seem to be missing this
Yes... and no...
There are some cases where jumbo frames saves the day, say for example on a SDN where you have an overlay made of VXLANs... to enable MTU 1500 inside the virtual LANs, you'll need jumbo enabled on the physical transport network, otherwise the VXLAN overhead will "eat into" the maximum usable MTU of the "payload" traffic.
Same if you use any sort of tunneling/GRE/IPSec and what not, and need to carry 1500byte packets.
Soooo.. no, this time around I'll disagree with your opinion. I agree btw that in a homelab their use is rarely justifiable, but in a broader scenario, they have their places.
Oh sure, increased MTU for tunneling and encapsulation is a whole other ballgame ( and of course unavoidable). When I hear and refer to "jumbo frames" I'm talking about just blindly cranking everything to 9000 in hopes to speed up transfers and crap like we used to have to do in 2010
Just trim off the version number for more recent manuals:
Didn't link to it though as my understanding was the last version suitable was 8.0.30(u, from memory). Can newer versions be run on the 6610 and 6650?
Index of /fastiron
docs.ruckuswireless.com
Didn't link to it though as my understanding was the last version suitable was 8.0.30(u, from memory). Can newer versions be run on the 6610 and 6650?
In my case, nearly all of that happens in virtual switching within the servers. Where it doesn't, they're on the IB switch with the SAN and can talk to each other over that. The only reason for jumbo on Eth was to speed up the access layer. Not worth the pain in this case, would rather have reliable OOB management than 5 odd percent or whatever improvement that one might reap in terms of goodput.
As noted, inband worked after setting the vlan ip mtu, if that takes your fancy, but I'd rather reduce the chances of remote throat cutting.
Latest 6 series will run is 8030 indeed. That link has the first revision of the 8030 manuals up to the b patch release. The latest manuals direct from ruckuses site are 8030h if I remember right, and include details about new commands etc added between the initial 8030 release and now. Those are what's included in the firmware zip
I don't love it.. but at least for the interim, I put a rate limit on the interface. This solves the critical need of not destroying skype/zoom calls at inappropriate times.
Code:
#int e 1/1/20
#rate-limit in fixed 500000
Rate Limiting on Port 1/1/20 - Config: 500000 kbps, Actual: 500000 kbps.I haven't played with any QoS related stuff on the Brocade as yet, but generally you should be able to tag traffic on ingress in to a switch interface from the source. You then shape (or at least, police) the tagged traffic when it hits another interface.
If you are lucky, the Brocade would support shaping/policing on egress traffic as it exits the switch on the LAG to the FW. If not, you have to rely on the shaping/policing (based on the tags applied to the traffic on ingress from the source to the switch) of the ingress interfaces of the next device (i.e. your firewall in this case). Most firewalls support some form of QoS, including pfsense (though never tried doing it on a LAG, just single interfaces). OPNsense is a pfSense fork, so it should support something similar. Whether you are doing it at layer 2 or layer 3 wouldn't really matter since your not controlling a particular type of traffic, just anything and everything source from a specific interface (i.e. camera networks switch interface).
pfSense claim it can handle up to a few Gbps when routing on average PC hardware. I've never seen that much from it personally, but last time I tested it (5 years ago), it was with hardware that was already about 4 years old at that time (~600mhz firewall box). It crapped out cpu bound at about 750mbps. If you need more than that, maybe a dedicated routing distro is the way to go, or else look at Netgate's TNSR.
If you are super keen, you could try rolling your own with DPDK and Vector Packet Routing etc, which is basically what is providing the performance enhancements to TNSR over pfSense. Personally, I'd just pay for TNSR rather than attempt to DIY it.
An alternative would be to look at doing in virtualised switching, OpenVSwitch comes to mind as the simplest to setup, probably with sufficient capability to do what you want with minimal overheads.
Last edited:
Hey guys newbie here.
Thanks to this amazing thread and everyone here i saved almost a grand and instead of buying the Ubiquity 48 Pro Switch i went with the 6450-48p
I still have a long time before the switch comes here and i am starting to think of how i am going to quiet this baby.
Has anyone here attempted a 12 o 14 inch fan mod on top of the switch ?
I know i have a ton of them from my pcs good quality fans used also for watercooling and static pressure is king i still plan on ordering some MB40201VX in case my plan doesn't go as well.
Im curious to know your thoughts on this
Thanks to this amazing thread and everyone here i saved almost a grand and instead of buying the Ubiquity 48 Pro Switch i went with the 6450-48p
I still have a long time before the switch comes here and i am starting to think of how i am going to quiet this baby.
Has anyone here attempted a 12 o 14 inch fan mod on top of the switch ?
I know i have a ton of them from my pcs good quality fans used also for watercooling and static pressure is king i still plan on ordering some MB40201VX in case my plan doesn't go as well.
Im curious to know your thoughts on this
Have you tried
mtu-exceed forward in your configuration? I believe this changes the switch's default behavior of dropping packets larger than a port's MTU, so that it will instead fragment and pass them along.Personally I feel that whoever even suggests or asks about implementation of Jumbo Frames in any way, needs to be violently and firmly shutdown.
UGH, i'm still putting out weird Vmware distribution switch warnings months have removing Jumbo frames from the systems.
I wasn't able to see any advantage from going with MTU 9000 and I rue the day that I decided to try it out in our dev environment.
UGH, i'm still putting out weird Vmware distribution switch warnings months have removing Jumbo frames from the systems.
I wasn't able to see any advantage from going with MTU 9000 and I rue the day that I decided to try it out in our dev environment.