I finally got to spend some quality time with my Arista 7050QX-32 and Azure Sonic. Sonic, out of the box without any configuration whatsoever, has BGP running on all of the interfaces. This is intended to allow Layer 3 native IP networking for each of the hosts and their containers, significantly simplifying networking without the typical L2 container networking overlays. After installing and configuring my hosts for BGP, I can get TCP performance at line rate between containers on different hosts:
$ docker run -it --name mycentos centos bash
[root@1e24805a735c /]# yum install iperf
...
[root@1e24805a735c]# iperf -c 192.168.120.2
------------------------------------------------------------
Client connecting to 192.168.120.2, TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.111.2 port 34042 connected with 192.168.120.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 24.7 GBytes 21.2 Gbits/sec
[root@1e24805a735c]# iperf -c 192.168.120.2 -P 4
------------------------------------------------------------
Client connecting to 192.168.120.2, TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[ 6] local 192.168.111.2 port 34050 connected with 192.168.120.2 port 5001
[ 4] local 192.168.111.2 port 34046 connected with 192.168.120.2 port 5001
[ 3] local 192.168.111.2 port 34044 connected with 192.168.120.2 port 5001
[ 5] local 192.168.111.2 port 34048 connected with 192.168.120.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 10.8 GBytes 9.27 Gbits/sec
[ 4] 0.0-10.0 sec 10.6 GBytes 9.12 Gbits/sec
[ 3] 0.0-10.0 sec 10.6 GBytes 9.13 Gbits/sec
[ 5] 0.0-10.0 sec 11.2 GBytes 9.60 Gbits/sec
[SUM] 0.0-10.0 sec 43.2 GBytes 37.1 Gbits/sec
...
L3 networking has the advantage of fewer proprietary extensions that enable large cloud-like fabrics while allowing dynamic config via standard routing software and isolating tenants. (that's the sales pitch anyways, in buzzword lingo)
I'll post/spam my experiences installing it while it's still fresh.
$ docker run -it --name mycentos centos bash
[root@1e24805a735c /]# yum install iperf
...
[root@1e24805a735c]# iperf -c 192.168.120.2
------------------------------------------------------------
Client connecting to 192.168.120.2, TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.111.2 port 34042 connected with 192.168.120.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 24.7 GBytes 21.2 Gbits/sec
[root@1e24805a735c]# iperf -c 192.168.120.2 -P 4
------------------------------------------------------------
Client connecting to 192.168.120.2, TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[ 6] local 192.168.111.2 port 34050 connected with 192.168.120.2 port 5001
[ 4] local 192.168.111.2 port 34046 connected with 192.168.120.2 port 5001
[ 3] local 192.168.111.2 port 34044 connected with 192.168.120.2 port 5001
[ 5] local 192.168.111.2 port 34048 connected with 192.168.120.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 10.8 GBytes 9.27 Gbits/sec
[ 4] 0.0-10.0 sec 10.6 GBytes 9.12 Gbits/sec
[ 3] 0.0-10.0 sec 10.6 GBytes 9.13 Gbits/sec
[ 5] 0.0-10.0 sec 11.2 GBytes 9.60 Gbits/sec
[SUM] 0.0-10.0 sec 43.2 GBytes 37.1 Gbits/sec
...
L3 networking has the advantage of fewer proprietary extensions that enable large cloud-like fabrics while allowing dynamic config via standard routing software and isolating tenants. (that's the sales pitch anyways, in buzzword lingo)
I'll post/spam my experiences installing it while it's still fresh.