Azure Sonic on the Arista 7050QX-32

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
A

AT S37=0

Member
Jul 12, 2016
56
22
8
58
Waiting for kijiki to give me a community edition
injinj.github.io
I finally got to spend some quality time with my Arista 7050QX-32 and Azure Sonic. Sonic, out of the box without any configuration whatsoever, has BGP running on all of the interfaces. This is intended to allow Layer 3 native IP networking for each of the hosts and their containers, significantly simplifying networking without the typical L2 container networking overlays. After installing and configuring my hosts for BGP, I can get TCP performance at line rate between containers on different hosts:

$ docker run -it --name mycentos centos bash
[root@1e24805a735c /]# yum install iperf
...

[root@1e24805a735c]# iperf -c 192.168.120.2
------------------------------------------------------------
Client connecting to 192.168.120.2, TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.111.2 port 34042 connected with 192.168.120.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 24.7 GBytes 21.2 Gbits/sec

[root@1e24805a735c]# iperf -c 192.168.120.2 -P 4
------------------------------------------------------------
Client connecting to 192.168.120.2, TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[ 6] local 192.168.111.2 port 34050 connected with 192.168.120.2 port 5001
[ 4] local 192.168.111.2 port 34046 connected with 192.168.120.2 port 5001
[ 3] local 192.168.111.2 port 34044 connected with 192.168.120.2 port 5001
[ 5] local 192.168.111.2 port 34048 connected with 192.168.120.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 10.8 GBytes 9.27 Gbits/sec
[ 4] 0.0-10.0 sec 10.6 GBytes 9.12 Gbits/sec
[ 3] 0.0-10.0 sec 10.6 GBytes 9.13 Gbits/sec
[ 5] 0.0-10.0 sec 11.2 GBytes 9.60 Gbits/sec
[SUM] 0.0-10.0 sec 43.2 GBytes 37.1 Gbits/sec

...

L3 networking has the advantage of fewer proprietary extensions that enable large cloud-like fabrics while allowing dynamic config via standard routing software and isolating tenants. (that's the sales pitch anyways, in buzzword lingo)

I'll post/spam my experiences installing it while it's still fresh.
 
  • Like
Reactions: larryexchange and Patrick
A

AT S37=0

Member
Jul 12, 2016
56
22
8
58
Waiting for kijiki to give me a community edition
injinj.github.io
Installation

1. Get into aboot from a tty.

I used kermit:
(/home/admin/) C-Kermit>set line /dev/ttyS0
(/home/admin/) C-Kermit>set speed 9600
(/home/admin/) C-Kermit>conn

Boot the switch and hit ctrl-c at the prompt. You should have a about shell at this point.

2. Copy the Sonic distribution for Arista from here:
Supported Devices and Platforms · Azure/SONiC Wiki · GitHub

I put this on a USB stick and inserted into the switch. Aboot will mount the USB as /mnt/usb1 and the flash as /mnt/flash.

Backup the existing EOS software onto the USB, it will be erased by the Sonic install, there will be no backup saved on the switch.

# cd /mnt/flash
# cp -a ./* /mnt/usb1/
# cp /mnt/usb1/sonic-aboot-broadcom.swi .

3. Boot Sonic

# boot /mnt/flash/sonic-aboot-broadcom.swi

This erases the flash and installs Sonic onto it. You clear the flash again and boot EOS by following the aboot instructions here:
EOS Section 6.5: Aboot Shell - Arista

I used 'fullrecover' and 'boot EOS-*.swi' several times. The EOS, boot-config and startup-config can be restored from the USB drive.

4. Login to sonic:

login: admin
passwd: YourPaSsWoRd

The eth0 device is the gigabit management interface, it will start dhclient to get the address. I usually setup my /etc/dhcp/dhcpd.conf to hand out addresses based on the MAC address (use 'ip addr show eth0' in the tty):

host arista {
option host-name "arista.local";
hardware ethernet 44:4c:a8:12:87:c6;
fixed-address 10.3.2.63;
}

After eth0 is configured, you can ssh admin@10.3.2.63 and skip the slow tty.
 
A

AT S37=0

Member
Jul 12, 2016
56
22
8
58
Waiting for kijiki to give me a community edition
injinj.github.io
5. Sonic does not dynamically adjust the fans on this switch. It puts them at high speed, so it will be very annoying if the switch is anywhere close to you. You can read the sensors and adjust the fans manually, so there must be some software out there that does dynamically adjust them. I throttle the fans manually.

# echo 120 > /sys/devices/platform/sb800-fans/hwmon/hwmon1/pwm1
# echo 120 > /sys/devices/platform/sb800-fans/hwmon/hwmon1/pwm2
# echo 120 > /sys/devices/platform/sb800-fans/hwmon/hwmon1/pwm3
# echo 120 > /sys/devices/platform/sb800-fans/hwmon/hwmon1/pwm4
# cat /sys/devices/platform/sb800-fans/hwmon/hwmon1/fan1_input
14489

The command 'show' is the cli way of examining the state of the switch:

# show environment
Command: sudo sensors
k10temp-pci-00c3
Adapter: PCI adapter
Cpu temp sensor: +36.0 C (high = +70.0 C)

fans-isa-0000
Adapter: ISA adapter
fan1: 12611 RPM
fan2: 12611 RPM
fan3: 12611 RPM
fan4: 12381 RPM

** Update:

There is a fancontrol daemon, but it needs configuration via pwmconfig. I think this needs to be done in the pmon container, at least that works for me.

# docker exec -it pmon bash
# pwmconfig

"pwmconfig" will allow you to choose which temp sensor devices control which fans. The 'fancontrol' daemon doesn't quite agree with "pwmconfig" for some temp sensors, so I used the CPU temperature to control the fan speed, since it is the highest value.

Test it:

# VERBOSE=1 /etc/init.d/fancontrol start

It should start automatically on boot by /usr/bin/pmon.sh when it exists in the platform config /usr/share/sonic/device/x86_64-arista_7050_qx32/fancontrol

6. Exploring Sonic

The architecture of Sonic is described here:

Architecture · Azure/SONiC Wiki · GitHub

It is a collection of daemons which use a Redis database to communicate and persist configuration. You can use 'redis-cli' to examine the database and the activity of the daemons. For example, this shows what the daemons are doing:

# redis-cli psubscribe '*'

The file /etc/sonic/config_db.json contains the startup configuration. The Redis database is populated from this and the daemons configure the switch by using Redis operations. OpenSwitch also has foundations for a Redis based configuration, but it seems a bit behind, whereas Microsoft is building a new Sonic version daily (Artifacts of buildimage-brcm-aboot-all #274 : /target [Jenkins]).

The daemons in Sonic are inside docker containers. For example, the BGP container looks like this:

# docker ps -a
...
224f774178f0 docker-fpm-quagga:latest "/usr/bin/supervisord" 4 days ago Up 4 days bgp
...

# docker exec -it bgp bash
# ps ax
PID TTY STAT TIME COMMAND
1 ? Ss+ 2:16 /usr/bin/python /usr/bin/supervisord
29 ? S 0:00 python /usr/bin/bgpcfgd
33 ? Sl 0:03 /usr/sbin/rsyslogd -n
38 ? S 0:02 /usr/lib/quagga/zebra -A 127.0.0.1
40 ? S 1:26 /usr/lib/quagga/bgpd -A 127.0.0.1 -F
42 ? Sl 0:00 fpmsyncd
180 ? Ss 0:00 bash
184 ? R+ 0:00 ps ax

# cat /etc/quagga/bgpd.conf
...
router bgp 65100
bgp log-neighbor-changes
bgp bestpath as-path multipath-relax
no bgp default ipv4-unicast
bgp router-id 10.1.0.32
 
Last edited:
A

AT S37=0

Member
Jul 12, 2016
56
22
8
58
Waiting for kijiki to give me a community edition
injinj.github.io
7. Setting up the IP addresses

As it is installed in build #270, Sonic defines each of the 32 ports with a with an EthernetX device and a 10.0.0.Y address, where X = (port-number-1) * 4 and Y = (port-number-1) * 2 (where port-number is 1 -> 32).

# ip addr show | grep inet
...
inet 10.0.0.0/31 brd 255.255.255.255 scope global Ethernet0
inet 10.0.0.2/31 brd 255.255.255.255 scope global Ethernet4
inet 10.0.0.4/31 brd 255.255.255.255 scope global Ethernet8
...

The address to use when you connect to these ports is 10.0.0.Z/31, where Z is (port-number-1) * 2 + 1. The first 16 ports use internal BGP, the second 16 ports are external BGP. Since external BGP is easier to configure than internal, I used the ports 17 -> 32 (I plan on creating a L2 lan with the first 16 ports anyway).

The addresses assigned to the hosts are

port 17 = 10.0.0.33/31
port 18 = 10.0.0.35/31
port 19 = 10.0.0.37/31
...

Since the layer 2 network set up using this config is confined to the link between the switch and the port, the host has to route through the switch in order to connect with the other ports otherwise any IP address other than the link pair will try to route to the wrong network. I did this by setting up my router as the default-gateway, which propagates default route through the switch to the hosts.

Install quagga and start zebra:
# yum install quagga
# systemctl start zebra

My 40gbe devices are named fo0 and fo1.

# vtysh
router# config term
router(config)# log file /var/log/quagga/quagga.log
router(config)# interface fo0
router(config-if)# ip address 10.0.0.49/31
router(config-if)# no shutdown
router(config-if)# end
router# write
Building Configuration...
Configuration saved to /etc/quagga/zebra.conf

You should be able to ping 10.0.0.48 now, which is port 25 on the switch.
# ping 10.0.0.48
PING 10.0.0.48 (10.0.0.48) 56(84) bytes of data.
64 bytes from 10.0.0.48: icmp_seq=1 ttl=64 time=0.198 ms

One other thing I did on the router was route the 10.0.0.0/26 range through the switch:

# ip route add 10.0.0.0/26 via 10.0.0.48 dev fo0

Repeat the above for each host connected, except for the last line.
 
A

AT S37=0

Member
Jul 12, 2016
56
22
8
58
Waiting for kijiki to give me a community edition
injinj.github.io
8. Setting up BGP routing

Each of my hosts has a docker0 device on it using a unique ip address range. BGP is going to distribute the routes for the docker0 brige across the cluster.

# cat /etc/docker/daemon.json
{
"dns": ["192.168.255.1"],
"iptables": false,
"ip-forward": false,
"ip-masq": false,
"storage-driver": "btrfs",
"bip": "192.168.37.1/24",
"fixed-cidr": "192.168.37.0/24",
"mtu": 1500
}

The dns entry is a caching named on the router. I set up a loopback device with this ip address and distributed it with BGP.

Here is the /etc/quagga/bgpd.conf for the above docker bridge:

# cat /etc/quagga/bgpd.conf
hostname myhost
password zebra
log file /var/log/quagga/bgp.log

router bgp 64005
bgp router-id 10.0.0.41
network 192.168.37.0/24
neighbor 10.0.0.40 remote-as 65100

Start bgpd
# systemctl start bgpd

The 64005 AS number above is unique for each port from 17 -> 32. These are configured be on the switch in the BGP container in /etc/quagga/bgpd.conf via the Redis database.

port 17 = 64001
port 18 = 64002
port 19 = 64003
...
port 21 = 64005

The above host is in port 21. After start bgpd on each of the hosts, you should see the routes populated by BGP.

On the switch:

root@arista:/# ip route show
default via 10.0.0.49 dev Ethernet96 proto zebra
...
192.168.37.0/24 via 10.0.0.41 dev Ethernet80 proto zebra
...

All the hosts are set up the same, except for the router. The router has this in the /etc/quagga/bgpd.conf:

hostname router
password zebra
log file /var/log/quagga/bgp.log
router bgp 64009
bgp router-id 10.0.0.49
network 192.168.255.0/24
neighbor 10.0.0.48 remote-as 65100
neighbor 10.0.0.48 default-originate

Note the "default-originate" line. This propagates the default route to the hosts and the switch.

I also block the BGP port from the internet side:
# iptables -A INPUT -i te0 -p udp -m udp --dport 179 -j DROP
# iptables -A INPUT -i te0 -p tcp -m tcp --dport 179 -j DROP
 
You must log in or register to reply here.
Top Bottom