Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[jode]

I'm having a bit of an issue with traffic flowing through the ICX6610.

What makes you think this is an issue with the ICX6610?

If I lower either the LAN or WAN interface down to 1gbe I get about 940mbps just fine.

A change in OpnSense changes the situation significantly. This points to an issue in OpnSense.

Anyone have any ideas what could cause this and a fix?

Is OpnSense managing the layer 3 routing in your network?

I assume OpnSense hw does not have enough single core performance to keep up with your network traffic and therefore becoming the bottleneck.

There are multiple threads explaining how to enable layer 3 routing in ICX6610. You can even find examples buried deep in this thread.

 

[Marc_]

Need a little help with a pair of ICX6610-48 I have coming. I want to stack them using the rear QSFP ports. My question is, will any transceiver work or are they brocade specific? I've got some old meraki stuff laying around (MA-CBL-40G50CM) and would like to recycle them if possible?

EDIT: Can confirm these work

 

[autoturk]

Anybody have any first-hand experience with the icx7450-24p? I'm seeing a lot of them on ebay and having 40gb + 10gb copper + 10gb SFP+ seems like a home run feature wise, but don't know much about noise + power consumption

 

[manki09]

What makes you think this is an issue with the ICX6610?



A change in OpnSense changes the situation significantly. This points to an issue in OpnSense.



Is OpnSense managing the layer 3 routing in your network?

I assume OpnSense hw does not have enough single core performance to keep up with your network traffic and therefore becoming the bottleneck.

There are multiple threads explaining how to enable layer 3 routing in ICX6610. You can even find examples buried deep in this thread.

Thank you for replying Jode.

The ICX6610 is doing all of the routing in my network. My OPNSense (172.16.0.1/29) is unaware of any VLANs and simply has static routes for my 2 Vlans directed to the Brocade (172.16.0.2). The brocade then has a default gateway to OPNSense.

The reason why I think it's the brocade is when I connect a 2.5 or greater device direct to OPNSense using a spare IP on my 172.16.0.0/29 network I consistently get my full internet speeds (~2100/350). If I connect the brocade back up so that the only thing between the router and device is the brocade and on the same vlan the speeds tank down to ~550/350 with the same device connected at 1GB. I do also have a server connected up at 40gbps and it gets about 1000-1200mbps.



When I check the core utilization with htop or top on OPNSense. It never shows any core anywhere near 100% and they are generally 50% or lower. I've also tested OPNSense on another device that is 30% faster single core and double the multicore speed. That different hardware got exactly the same result. No higher, no lower speed.

When I do a speedtest with OPNSense set to 1GB I get no dropped packets on the Brocade, but when I turn change it back to 2.5G/10G the brocade starts to drop packets. This happens on all devices 1GB or higher. Note: I haven't tested conected at 100mbps or lower.

I attached 3 pictures in different senerios. Every test I made sure nothing had any resources maxed out down to individual cores.

In my opinion is seems that the brocade is being overloaded.... some how... and is dropping packets. This could also be a misconfiguration issue but all I have it doing is very simple static routing, no where near what it can handle.

 

[manki09]

Need a little help with a pair of ICX6610-48 I have coming. I want to stack them using the rear QSFP ports. My question is, will any transceiver work or are they brocade specific? I've got some old meraki stuff laying around (MA-CBL-40G50CM) and would like to recycle them if possible?

My experience with twinax cable is that they generally more universal than Fiber SFPs. I would recommend seeing if your Meraki ones work since you already have them.

 

[fohdeesha]

Thanks for the guides! They've been really helpful on setting up.

I have one question however, I did the licensing for a ICX7250, but noticed it said it's a trial for 45 days. Is this something I should be worried about?

its not a trial it will last forever

 

[jode]

Hi @manki09 ,

thanks for clarifying your network configuration.

When I do a speedtest with OPNSense set to 1GB I get no dropped packets on the Brocade, but when I turn change it back to 2.5G/10G the brocade starts to drop packets.

This sounds like an issue with the connection between the OPNSense and the Brocade. Is there a way you can try different cables/transceivers?

 

[manki09]

Hi @manki09 ,

thanks for clarifying your network configuration.



This sounds like an issue with the connection between the OPNSense and the Brocade. Is there a way you can try different cables/transceivers?

Hello, @jode

That was another thought I had.

My original setup was running a SFP+ RJ45 on the brocade to the 1/2.5/5/10g intel x550-T2 on the OPNSense. So I would have a 10G link between the two.

During testing I threw in a cheap 2x 10G SFP+ / 4x 2.5G RJ45 switch, which I had tested standalone, to eliminate the RJ45 SFP. I connected TwinAX from Brocade to cheap switch, then 2.5gb Ethernet to OPNSense. All negotiated correctly or set manually. (Intel x550-T2 NBase-T (2.5/5) has to be set manually). But still the problem persisted with the same results.

Yesterday, I also decided to test another SFP+ port on the brocade and still the problem persisted.



I've been toying with the idea of getting a used Cisco 3850 off ebay but want to get this working correctly since I already have it and is the cheaper option.

 

[jode]

All negotiated correctly or set manually. (Intel x550-T2 NBase-T (2.5/5) has to be set manually). But still the problem persisted with the same results.

Can you try a different cable (cat 6a)? Is there a way you can test with iperf3? Can you try a different SFP+ port on the ICX?

 

[manki09]

Can you try a different cable (cat 6a)? Is there a way you can test with iperf3? Can you try a different SFP+ port on the ICX?

Just FYI on that intel NIC they disabled 2.5g and 5g from being advertised/negotiated to. I don't think there was a specific reason other than 2.5/5g devices weren't common when released

I have tried several different cables. The length of all the cables between the modem, router and brocade are less then 1 meter.

I have tried a couple different SFP ports on the ICX with no luck.

I will try some iperf tests and see where that takes me.

I was able to acquire the same model ICX6610 that my work had and was basically not in use and will be trying to load my config onto that to eliminate hardware issue.

 

[Rttg]

the brocade starts to drop packets

There's something about dropping packets that feels unlikely.

Looking at your config, is there a reason why you have ``flow-control`` disabled for every interface? That looks a bit odd to me after comparing it to my switch configs.

 

[manki09]

There's something about dropping packets that feels unlikely.

Looking at your config, is there a reason why you have ``flow-control`` disabled for every interface? That looks a bit odd to me after comparing it to my switch configs.

Hey @Rttg, thanks for the reply.
I only have flow-control disabled for testing purposes... I should have removed it before posting it. I had the problem before I disabled it.

 

[manki09]

Can you try a different cable (cat 6a)? Is there a way you can test with iperf3? Can you try a different SFP+ port on the ICX?

Swapped in the spare 6610 and got the same results.

I performed a few Iperf3 test.
I did notice a trend while responding to this and I've confirmed that this is not an OPNSense issue since I'm dropping packets to other devices besides OPNSense. Configuring the interface at 1GB was just a band-aid.


It seems to be dropping packets only when sending data from a higher speed interface to a lower. A couple examples below.


40GB to 10GB = dropped packets
40GB to 1GB = dropped packets

1GB to 10GB = no dropped packets
1GB to 40GB = no dropped packets
1GB to 1GB = no dropped packets

I also noticed dropped packets on the brocade interface the router is connected to while doing the upload portion during a internet speedtest. Which is limited at 350~.

 

[DouglasteR]

Seems like the CPU isn´t handling high speeds

 

[fohdeesha]

Seems like the CPU isn´t handling high speeds

cpu on what? traffic on the ICX is not handled by a CPU

if you're actually pushing (or trying to push) more than 10gb or 1gb to a 10gb or 1gb port, then you're going to drop packets - it's physics. If you have flow control enabled the switch can try to send pause frames to the sending host saying the receiving port/client can't receive that fast, but flow control is hit and miss. then you have other complications like the ASIC buffer which is gunna fill during this, and a bunch of other stuff. sending data from a fast port to a slow port is not gunna turn the slow port into a fast one, it can only fit so many packets

this sounds like weird throttling/tcp window scaling/etc on the opnsense side, especially if setting the opnsense interface to 1gbe so it can't send any faster than that fixes it

 

[manki09]

cpu on what? traffic on the ICX is not handled by a CPU

if you're actually pushing (or trying to push) more than 10gb or 1gb to a 10gb or 1gb port, then you're going to drop packets - it's physics. If you have flow control enabled the switch can try to send pause frames to the sending host saying the receiving port/client can't receive that fast, but flow control is hit and miss. then you have other complications like the ASIC buffer which is gunna fill during this, and a bunch of other stuff. sending data from a fast port to a slow port is not gunna turn the slow port into a fast one, it can only fit so many packets

this sounds like weird throttling/tcp window scaling/etc on the opnsense side, especially if setting the opnsense interface to 1gbe so it can't send any faster than that fixes it

@fohdeesha
Thank you for replying.
I understand that I wont get the higher speed while sending to a slower port but if it's dropping packets and I'm getting ~%20 slower rate than the slowest port is getting then there's something wrong.

I had an ASUS RT-89AX sitting around (Has a 10Gbase-T port and a 10GB SFP+ port). Both can be used for WAN or LAN. However with OPNSense out of the picture and with completely different hardware and software the speed test results did not change.

 

[kapone]

@fohdeesha
Thank you for replying.
I understand that I wont get the higher speed while sending to a slower port but if it's dropping packets and I'm getting ~%20 slower rate than the slowest port is getting then there's something wrong.

I had an ASUS RT-89AX sitting around (Has a 10Gbase-T port and a 10GB SFP+ port). Both can be used for WAN or LAN. However with OPNSense out of the picture and with completely different hardware and software the speed test results did not change.

There's something in your hardware/config that's not quite right.

I run a 6610 and Opnsense as well (Opnsense is on bare-metal with a 10gb connection to the 6610). Although my internet is only 1gb symmetric, I think given that our hardware/software is almost similar, it's relevant. I can do almost wire speed NAT from any machine on my network, regardless of its connection to the 6610 (I've tested with both 1gb and 10gb links.)



But...I don't have any 2.5g devices/adapters in the mix. Not sure if that's what might be an issue here.

 

[manki09]

I've got some good new. Just browsing around in the cli I found the symmetric-flow-control enable command. After I enabled it my speedtests on 1gb links shot up to where they should be ~940. At least on PCs with 1G NICs. I still need to test speeds more than 1GB, but I did notice there weren't any dropped packets too.

 

[pwm80211]

I posted a question in the general Network forum but I think this is the topic I should have asked the question originally. Can I set the ICX-6450 GUI to be DHCP?

 

[kpfleming]

Can I set the ICX-6450 GUI to be DHCP?

The 'GUI' doesn't have any configuration, you probably mean the management interface which is typically reached via VLAN 1 (using a virtual ethernet interface on that VLAN). If that's what you mean, yes, you can configure the VE on VLAN 1 to use DHCP to get address/gateway/etc. information, although that means if your DHCP server is unavailable for any reason you may lose the ability to manage the ICX over the network.