Brocade ICX switch performing worse than router

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Damo

Active Member
Sep 7, 2022
115
32
28
I have an issue with my router having significantly better speeds than my Brocade ICX6650 switch. Router > 1/1/41. iperf3 onto the router reaches full 10G speeds. What could the issue be?

Code:
root@router:~$ iperf3 -c lon.speedtest.clouvider.net -p 5205
Connecting to host lon.speedtest.clouvider.net, port 5205
[  5] local 193.xxx.xxx.1 port 44488 connected to 5.180.211.133 port 5205
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   412 MBytes  3.46 Gbits/sec  737   5.08 MBytes
[  5]   1.00-2.00   sec   439 MBytes  3.68 Gbits/sec  754   5.16 MBytes
[  5]   2.00-3.00   sec   356 MBytes  2.99 Gbits/sec  183   4.30 MBytes
[  5]   3.00-4.00   sec   465 MBytes  3.90 Gbits/sec  264   4.39 MBytes
[  5]   4.00-5.00   sec   472 MBytes  3.96 Gbits/sec   52   3.66 MBytes
[  5]   5.00-6.00   sec   488 MBytes  4.09 Gbits/sec  417   4.44 MBytes
[  5]   6.00-7.00   sec   481 MBytes  4.04 Gbits/sec  204   4.18 MBytes
[  5]   7.00-8.00   sec   535 MBytes  4.49 Gbits/sec    7   3.97 MBytes
[  5]   8.00-9.00   sec   526 MBytes  4.41 Gbits/sec    1   4.32 MBytes
[  5]   9.00-10.00  sec   474 MBytes  3.97 Gbits/sec  605   4.19 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.54 GBytes  3.90 Gbits/sec  3224             sender
[  5]   0.00-10.00  sec  4.54 GBytes  3.90 Gbits/sec                  receiver

iperf Done.
Following ran from a server connected to switch

Code:
root@test:~# iperf3 -c lon.speedtest.clouvider.net -p 5205
Connecting to host lon.speedtest.clouvider.net, port 5205
[  5] local 193.xxx.xxx.11 port 43054 connected to 5.180.211.133 port 5205
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  95.4 MBytes   800 Mbits/sec   32    457 KBytes      
[  5]   1.00-2.00   sec  65.6 MBytes   551 Mbits/sec  122    155 KBytes      
[  5]   2.00-3.00   sec  40.6 MBytes   341 Mbits/sec    0    286 KBytes      
[  5]   3.00-4.00   sec  38.9 MBytes   326 Mbits/sec   15    174 KBytes      
[  5]   4.00-5.00   sec  43.3 MBytes   363 Mbits/sec    0    306 KBytes      
[  5]   5.00-6.00   sec  52.1 MBytes   437 Mbits/sec    9    336 KBytes      
[  5]   6.00-7.00   sec  62.3 MBytes   523 Mbits/sec    1    346 KBytes      
[  5]   7.00-8.00   sec  65.5 MBytes   550 Mbits/sec    3    339 KBytes      
[  5]   8.00-9.00   sec  55.3 MBytes   464 Mbits/sec   62    362 KBytes      
[  5]   9.00-10.00  sec  59.5 MBytes   499 Mbits/sec    2    371 KBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   578 MBytes   485 Mbits/sec  246             sender
[  5]   0.00-10.00  sec   577 MBytes   484 Mbits/sec                  receiver

iperf Done.
---

Code:
SSH@6650switch>sh run
Current configuration:
!
ver 08.0.30uT323
!
stack unit 1
  module 1 icx6650-64-56-port-management-module
  module 2 icx6650-64-4-port-160g-module
  module 3 icx6650-64-8-port-80g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
fpod-40g-enable group 1
fpod-40g-enable group 2
fpod-40g-enable group 3
mtu-exceed forward
aaa authentication web-server default local
aaa authentication login default local
hostname 6650switch
ip dhcp-client disable
ip dns server-address 193.xxx.xxx.1
ip route 0.0.0.0/0 193.xxx.xxx.1
!
no telnet server
username root password .....
!
!
web-management https
!
!
!
!
!
!
!
interface ethernet 1/1/56
speed-duplex 1000-full-master
!
interface ethernet 1/2/1
disable
!
interface ve 1
ip address 10.0.10.51 255.255.255.0
!
!
!
!
!
!
!
!
!
end

SSH@6650switch>
---

Code:
SSH@6650switch#show interfaces ethernet 1/1/41
10GigabitEthernet1/1/41 is up, line protocol is up
  Port up for 47 minutes 16 seconds
  Hardware is 10GigabitEthernet, address is cc4e.2417.6180 (bia cc4e.2417.61a8)
  Interface type is unknown
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
  Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabl                                                                                                                                                                                                                                             ed
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 531480 bits/sec, 762 packets/sec, 0.00% utilization
  300 second output rate: 59570720 bits/sec, 5128 packets/sec, 0.60% utilization
  13265075 packets input, 8118964360 bytes, 0 no buffer
  Received 140849 broadcasts, 12 multicasts, 13124214 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  33400261 packets output, 47511414792 bytes, 0 underruns
  Transmitted 4352 broadcasts, 1468 multicasts, 33394441 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0            33400416                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   1                   0
    6                   0                   0
    7                   0                   0
 
Last edited:

Damo

Active Member
Sep 7, 2022
115
32
28
I'll mess around with the hardware just want to confirm the running config I have should be fine?
 

alsenior

Member Member
Apr 19, 2016
56
14
8
What does your topology look like? Is the port that the server is connected to running at 1G or 10G ?
 

Damo

Active Member
Sep 7, 2022
115
32
28
Can you post the output of 'sh int br' (feel free to trim out down ports).
Code:
Port    Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/2   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/3   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/4   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/5   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/6   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/7   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/8   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/9   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/10  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/11  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/12  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/13  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/14  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/15  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/16  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/17  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/18  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/19  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/20  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/21  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/22  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/23  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/24  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/25  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/26  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/27  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/28  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/29  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/30  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/31  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/32  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/33  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/34  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/35  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/36  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/37  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/38  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/39  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/40  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/41  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/42  Up      Forward Full 10G   None  No  1    0   cc4e.2417.6180
1/1/43  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/44  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/45  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/46  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/47  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/48  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/49  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/50  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/51  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/52  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/53  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/54  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/55  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/1/56  Down    None    None None  None  No  1    0   cc4e.2417.6180
1/2/1   Disable None    None None  None  No  1    0   cc4e.2417.6180
1/2/2   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/2/3   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/2/4   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/3/1   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/3/2   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/3/3   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/3/4   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/3/5   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/3/6   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/3/7   Down    None    None None  None  No  1    0   cc4e.2417.6180
1/3/8   Down    None    None None  None  No  1    0   cc4e.2417.6180
mgmt1   Down    None    None None  None  No  None 0   cc4e.2417.6180

Port    Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1     Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2417.6180
 

LodeRunner

Active Member
Apr 27, 2019
546
228
43
Yeah switch showing everything at 10 G and the config you posted is about as simple as it can possibly be. And MTU on the server NIC is 1500? Since I don't see jumbo frames configured on the switch.
 

Damo

Active Member
Sep 7, 2022
115
32
28
Yeah switch showing everything at 10 G and the config you posted is about as simple as it can possibly be. And MTU on the server NIC is 1500? Since I don't see jumbo frames configured on the switch.
Yep 1500 tried with and without jumbo frames
 

LodeRunner

Active Member
Apr 27, 2019
546
228
43
Super weird. My go-to suspects are:
  • Mistmatched MTU (covered)
  • Flow control
  • client specific settings (offload, interrupt moderation)
  • weird fault in the optic/DAC?
The 6650 should be able to handle 10G on all ports at the same time. I know for the 6610, after licensing you have to set the port speed to 10G, but the interface info shows it negotiated 10G, so that's probably not it.

What's the reason for the "speed-duplex 1000-full-master" statement on 1/1/56? Given that 1/1/56 is down, that probably doesn't matter, just curious.

Is "lon.speedtest.clouvider.net" your server or a hosting provided server? When doing the iPerf to it from the server, is the traffic going server > switch> router > iPerf server? If that's the case, what is the router? The router may not be able to handle 10G throughput.