ICX switch with jumbo frames - management port failure

[sth2100]

So I enabled jumbo frames, and all is well. I enable jumbo frames on my pc and now I cannot access the management port on the two ICX switches I have just setup. Do I need to somehow enable jumbo frames on the management port as well?

I just switched back to 1500 MTU and the management web interface loaded perfectly. Switching back to 9000 and it fails again.

 

[sth2100]

I did try going into the management interface to attempt setting jumbo at that level, but the jumbo command appears to be a globally scoped command.

ICX6610-48P Router#conf t
ICX6610-48P Router(config)#int mana
  management   Management ethernet port
ICX6610-48P Router(config)#int management 1
ICX6610-48P Router(config-if-mgmt-1)#jumbo
System already in Jumbo Mode!
ICX6610-48P Router(config-if-mgmt-1)#

 

[altmind]

does ICX have anything similar to cisco management vrf?

 

[sth2100]

Not familiar with the Cisco products, so I can't say.

 

[fohdeesha]

it's 2022, don't use jumbo frames

 

[altmind]

@fohdeesha jumbo frames are getting more and more necessary in the era of 10g ports. its hard to saturate the link without them.

 

[fohdeesha]

@fohdeesha jumbo frames are getting more and more necessary in the era of 10g ports. its hard to saturate the link without them.

Huh? the opposite is true as CPUs get faster and faster, handling interrupts at 10gb/s with 1500byte frames is not nearly the workload it used to be. I don't know many places that still run 9000byte MTU in production outside of 100ge+ storage networks, hell even on 10 year old Xeon E5 v1 CPUs I can hit ~30gbps with 1500 byte frames. If you're on a PC from this decade and need 9000byte frames to approach 10gbps, something else is very wrong

 

[fohdeesha]

inside a VM, with a *single CPU core*, on a 9 year old host at default 1500 MTU:

root@dhcp:~# iperf3 -c 192.168.1.10
Connecting to host 192.168.1.10, port 5201
[  4] local 172.16.110.2 port 38270 connected to 192.168.1.10 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.06 GBytes  9.10 Gbits/sec    0    416 KBytes
[  4]   1.00-2.00   sec  1.07 GBytes  9.19 Gbits/sec    0    416 KBytes
[  4]   2.00-3.00   sec  1.07 GBytes  9.18 Gbits/sec    0    416 KBytes
[  4]   3.00-4.00   sec  1.07 GBytes  9.17 Gbits/sec    0    416 KBytes
[  4]   4.00-5.00   sec  1.06 GBytes  9.14 Gbits/sec    0    416 KBytes
[  4]   5.00-6.00   sec  1.07 GBytes  9.20 Gbits/sec    0    416 KBytes
[  4]   6.00-7.00   sec  1.05 GBytes  9.04 Gbits/sec    0    416 KBytes
[  4]   7.00-8.00   sec  1.07 GBytes  9.19 Gbits/sec    0    416 KBytes
[  4]   8.00-9.00   sec  1.07 GBytes  9.18 Gbits/sec    0    416 KBytes
[  4]   9.00-10.00  sec  1.06 GBytes  9.14 Gbits/sec    0    416 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.7 GBytes  9.15 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  10.7 GBytes  9.15 Gbits/sec                  receiver

 

[Drewy]

Jumbo frames are the work of the devil.

 

[sth2100]

Two points...

1. You guys are avoiding the question. Why is the management port getting treated special from everything else by dropping jumbo frames? (let's go with a multiple choice answer to get this one behind us...)
   A - you don't know and I've uncovered something new and exciting!
   B - I screwed up and didn't set some specific setting on the management port - now I get to learn something.
   C - My ICX6610 is broken and sad.
   D - None of the above
2. You're making a very good point about the jumbo frame usage. I have often wondered if jumbos were needed anymore if the underlying network hardware had some hardware offloading feature to reduce this 1500 byte burden on the CPU. Or if CPUs just reached a point where they had some specialized instruction set to more efficiently handle this.

I don't currently have my 10G setup yet, that's what I'm planning to do next, to try and use the 40G QSFP+ ports and the 10G SFP+ port with some new cards I'll put into my old hardware. And yes, my hardware is the older v1 2600 Xeons. Currently everything is running jumbo except my PC now, but I'll have to do some performance testing comparison to see how well that works with the 10G and 40G connections.

Fun stuff!

 

[klui]

I don't use jumbo frames but take a look at Terry Henry's video on the subject. He talks about an mtu-exceed command. Maybe you can play with that. I don't see a reason why one should be transferring huge amount of data on a management port. Maybe Brocade's management port doesn't support large MTUs.

 

[sth2100]

Thanks Klui - I watched that video to actual enable my jumbo frames.

I did set the mtu-exceeds flag to have the switch fragment the packets for ports that needed it. For some reason that doesn't seem to be applying to the management port. So if it cannot detect the management port is expecting 1500 mtu, then I don't know how it will know something else is able to handle jumbo or not.

In my mind, an analogy would be shipping a heavy (jumbo) box to someone. You don't know if that someone receiving the box will be strong enough to pick it up and carry it inside, or if it will be some old lady who can't manage it. Until the person opens their door and tries to "receive" the box, you don't know. The strong person picks up the jumbo box and is fine with it... the old lady drops it and leaves it on the porch. So how does the package man (switch) who's job is to only drop it off on the porch, know if he needs to unbox it and create several smaller boxes before he leaves?

 

[klui]

Not following your analogy. Besides, use the common denominator of 1500 MTU to remain compatible with more clients for management.

There's a reason why 400G switches still have gigabit management ports and some switches have 100 Mb ports.

 

[i386]

I just switched back to 1500 MTU and the management web interface loaded perfectly. Switching back to 9000 and it fails again.

What nics do you have?

Some vendors have text areas where you could type in any numbers you want for jumbo frames. Set a "wrong" number and you mess up your network because other devices can not handle these ethernet frames.

Other vendors (like intel) allow only certain/"more common" values for jumbo frames.

 

[sth2100]

Some vendors have text areas where you could type in any numbers you want for jumbo frames. Set a "wrong" number and you mess up your network because other devices can not handle these ethernet frames.

I'm using Linux so I set the MTU via ifconfig commands.

This is the first switch I've come across that does not support jumbo frames out of the box. Every other consumer grade switch I've used must default MTU to a large number as I've never had issues with jumbo frames except on the hosts themselves.

 

[sth2100]

There's a reason why 400G switches still have gigabit management ports and some switches have 100 Mb ports.

I'm not concerned with the throughput of the management port, I'm concerned with the compatibility of it. For instance, if I want to use jumbo frames (a feature offered by Brocade switches), I can use them so long as I don't want to manage the switch. The moment I want to manage the switch, I have to remember to set my MTU back to 1500, then remember to raise it back up to 9000 when I'm done or put added strain on the CPU (potentially - haven't actually run any perf tests on this).

Just seems odd that they would design such a great switch and overlook this when dealing with the management port. I can live with it, and may go back to 1500 MTU on everything, but it seems odd, or maybe they had a very good reason for doing it this way.

 

[fohdeesha]

This is working as designed, the isolated dedicated management port is running at 1500MTU, like the management port on every other enterprise switch out there. Those ports are for separate networks dedicated to management traffic, which would never be running anything but default MTU when designed properly. The real question is why are you using the isolated management port if you're still accessing it with your regular computer that's already on the main network? Just give the switch an IP in-band on a VE like my guide outlines and manage the switch over the main network you're already connected to. Management ports are for management networks

I'm not concerned with the throughput of the management port, I'm concerned with the compatibility of it.

as mentioned above, compatibility is the exact reason the dedicated MGMT port is stuck at 1500mtu. I've never seen a management network anywhere that runs anything but default MTU, the switch would become completely useless in 99% of installations if the management port started sending out 9000byte frames


And I can't overstate this: it's 2022 and you're only running 10/40gbe. Don't run jumbo frames. The only remaining use for them on networks this size this century is generating endless amounts of posts like this where stuff is inevitably broken

 

[fohdeesha]

I'm using Linux so I set the MTU via ifconfig commands.

This is the first switch I've come across that does not support jumbo frames out of the box. Every other consumer grade switch I've used must default MTU to a large number

Keyword: consumer grade. The icx was a $6,000 L3 enterprise switch, and every L3 switch model I've worked with regardless of vendor defaults to 1500 MTU (because defaulting to 9000 would completely break 99% of networks the second you try to route something with it). This isn't a dumb L2 bridge like most consumer switches

 

[klui]

Reading how you manage your switch implies you're not using best practice as @fohdeesha wrote above. Management networks are supposed to be isolated from the data network. If your data network is compromised, how would you use it to manage the network? Either use another NIC, or a tagged VLAN on your client.

 

[sth2100]

Completely agree with what you guys are saying... I am learning quite a bit here from feedback. I see your points on the 1500 MTU and I have encountered my own issues with it over the years when some things on the network are not setup properly and it does waste time tracking that down - I can only imagine the nightmare that would be in a large datacenter.

Had assumed the management port should be on it's own network, but I really didn't want to bother with that currently as I believe it's more setup headache than it's worth for a home lab setup. I will simply unplug the management port for now and only plug it in if I ever need to use it, or if I get adventurous and dive into vlan territory one day.

For now I'll listen to your warnings and switch everything back to 1500 MTU at the switch, and all devices, to keep things simple and to standard.