Combining two ICX6650 switches

[Harmony]

I have two ICX6650 switches that I want to connect to my Broadcom BCM57810S NIC (one SFP+ to each switch) however I am not sure if there is anything I need to change to get this working outside of the box. So far the only thing i've done is enable Jumbo frames do I need to configure Link Aggregation (802.3ad LACP) following ICX6xxx Advanced - Fohdeesha Docs ?

auto lo
iface lo inet loopback

auto enp38s0
iface enp38s0 inet manual

auto enp45s0f0np0
iface enp45s0f0np0 inet manual

auto enp45s0f1np1
iface enp45s0f1np1 inet manual

iface bond1 inet manual
bond-slaves enp45s0f0np0 enp45s0f1np1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
up ip link set enp45s0f0np0 master bond1
up ip link set enp45s0f1np1 master bond1
mtu 9000

auto vmbr0
iface vmbr0 inet static
address 192.168.0.250/24
gateway 192.168.0.1
bridge-ports enp38s0
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 10.0.10.250/24
gateway 10.0.10.1
bridge-ports bond1
bridge-stp off
bridge-fd 0
mtu 9000

From the tests I have run (one below) it seems to me to be working smooth but I have been informed on other public discussions without much elaboration that "you can't put them all in a bond / bridge like that with stp off and connect them to two switches without problems".

iperf3 -c 10.0.10.202 -i 1 -V -u -b 1.4M

iperf 3.9
Linux BUDG1 5.15.74-1-pve #1 SMP PVE 5.15.74-1 (Mon, 14 Nov 2022 20:17:15 +0100) x86_64
Control connection MSS 8948
Setting UDP block size to 8948
Time: Fri, 25 Nov 2022 18:32:09 GMT
Connecting to host 10.0.10.202, port 5201
      Cookie: uhtsnvpw4iar7mdzvcw643vwu4lsqi2oydwk
      Target Bitrate: 1400000
[  5] local 10.0.10.200 port 38091 connected to 10.0.10.202 port 5201
Starting Test: protocol: UDP, 1 streams, 8948 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   175 KBytes  1.43 Mbits/sec  20
[  5]   1.00-2.00   sec   175 KBytes  1.43 Mbits/sec  20
[  5]   2.00-3.00   sec   166 KBytes  1.36 Mbits/sec  19
[  5]   3.00-4.00   sec   175 KBytes  1.43 Mbits/sec  20
[  5]   4.00-5.00   sec   166 KBytes  1.36 Mbits/sec  19
[  5]   5.00-6.00   sec   175 KBytes  1.43 Mbits/sec  20
[  5]   6.00-7.00   sec   166 KBytes  1.36 Mbits/sec  19
[  5]   7.00-8.00   sec   175 KBytes  1.43 Mbits/sec  20
[  5]   8.00-9.00   sec   166 KBytes  1.36 Mbits/sec  19
[  5]   9.00-10.00  sec   175 KBytes  1.43 Mbits/sec  20
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.67 MBytes  1.40 Mbits/sec  0.000 ms  0/196 (0%)  sender
[  5]   0.00-10.04  sec  1.67 MBytes  1.40 Mbits/sec  0.068 ms  0/196 (0%)  receiver
CPU Utilization: local/sender 1.1% (0.2%u/0.9%s), remote/receiver 0.0% (0.0%u/0.0%s)

iperf Done.

 

[NablaSquaredG]

I think you also need to configure MCT on the switches (Multi Chassis Trunking)

Configuring MCT

docs.ruckuswireless.com

 

[LodeRunner]

The ICX6650 doesn't support stacking, so you'll have to setup MCT as indicated by NablaSquaredG, or you will eventually have issues, either with traffic not flowing, or more likely, packet loss that doesn't make sense.

Also, with an iperf test clamped at T1 speed (1.4 Mbits? what kind of test is that?) you'll never know if your LACP (802.3ad) link is working properly, since that's a single flow well below the max bandwidth. You'll need multiple flows to force it to actually try to use both links.

MCT or not, with multiple, non-stacked switches, you want spanning tree. It will help save your sanity when you accidentally create a loop or misconfigure a LACP bundle between them.