ONIE switch

[AT S37=0]

Anybody running Cumulus, OpenNetLinux, OpenSwitch, or other white box/SDN offering?

I just got a Dell S6000-on in my lab and am playing with OpenSwitch. It's quite exhilarating to be able to use ports as if they were Linux ethernet devices.

So far I've been able to connect a bunch of Mellanox connectX3 cards to it using cheap TwinAX and yellow Fibre, both with Mellanox transceivers. Configuring layer 2 is exactly the same as Linux, for example creating VLAN 10 is defined by creating a bridge and adding a link dev tagged with vlan 10 to it.

$ brctl addbr mybridge

$ ip link add link e101-001-0 name myeth type vlan id 10

$ brctl addif mybridge myeth

I'm curious about the other offerings.

 

[Patrick]

@AT S37=0

Personally, I am ultra interested in this project. I would love to see more about it.

 

[AT S37=0]

It looks like OpenSwitch uses 32 Linux TAP devices for the ports and interfaces these with the SAI (Switch Abstract Interface) as defined by Facebook's OpenCompute, in the kernel. Many of the switch admin scripts are written in python, which has it's own interface with the control plane using C.

 

[AT S37=0]

Building openswitch (from GitHub - open-switch/opx-build). Needs docker and I have it setup for user level access (docker in /etc/group). I use centos 7 as my workstation dev environment with docker-ce packages installed.

$ sudo curl https://storage.googleapis.com/git-repo-downloads/repo > /usr/local/bin/repo
$ sudo chmod 755 /usr/local/bin/repo

# git clones the open switch packages from Dell's github repos:
$ mkdir opx ; cd opx
$ repo init -u git://git.openswitch.net/opx/opx-manifest && repo sync

# this creates a debian docker image for building
$ pushd opx-build/scripts ; ./opx_setup ; popd

# this runs a bash shell in the debian build environment (doesn't do any build operation)
$ ./opx-build/scripts/opx_run

# build 'all' repos using opx_run above and executing opx_build
$ ./opx-build/scripts/opx_run /bin/bash -ci "cd /mnt && opx-build/scripts/opx_build all"

# enter the docker image and package together to create a switch install image
$ ./opx-build/scripts/opx_run
$ cd /mnt
$ ./opx-build/scripts/opx_rel_pkgasm.py -b onie-opx-installer/release_bp/OPX_dell_base.xml -n 1

# the final product, the "installer" image (renamed to onie-installer) is what ONIE looks for on startup
$ ls PKGS*
PKGS_OPX-2.1.0.1-installer-x86_64.bin PKGS_OPX-2.1.0.1-x86_64-rootfs.tgz

 

[AT S37=0]

It looks like Cumulus is based on the same thing as OpenSwitch, just 3 years further along. A lot of what Cumulus does is open source and looks like anyone running OpenSwitch could use some of their stack: Cumulus Networks · GitHub

I point to the ifupdown package as proof of this. This is the Debian package responsible for bring up the network. It doesn't quite work automatically in OpenSwitch as far as I can tell, because the network control plane in OpenSwitch does not integrate with it directly. Instead, ifupdown communicates with the underlying Linux devices:

ifupdown -> linux tap device -> something something -> SAI

OPX control -> something something -> SAI

The result is that ifupdown doesn't see the switch ports until OPX control finishes initializing and that is asynchronous to ifupdown. This means that ifupdown works after the OPX control plane finishes, not during the Linux systemd init sequences, so it isn't able to initialize the switch during boot.

I believe this is the reason for this: GitHub - CumulusNetworks/ifupdown2

This is just a guess, I have not tried to incorporate ifupdown2 or determine how the underlying infrastructure of both packages differs.

 

[Biren78]

Your experience so far is it worth it or no?

 

[AT S37=0]

No regrets, even in its unfinished state you can work around things and get stuff to work. It's already served it's purpose, connect my rack at 40gb, there's nothing wrong with the packet forwarding... but then again, I only have a rack, not a datacenter mesh. It would be a sad day if this project is consumed by the networking goliaths.

Your experience so far is it worth it or no?

 

[Evan]

I would home we may see more of this, juts look at the Cisco nexus, it's based on Intel Xeon CPU, with even the big guys using more common hardware as a base an open switch software should in theory become easier to keep up if it can use similar hardware to everything else in the world and not so much custom ASIC etc

 

[AT S37=0]

One of the crazy side effects of using Intel CPUs and a Debian 8 base is that I can compile C/C++ programs on my HP CentOS 7 workstation and run them on the switch. No special Docker environment needed, the binaries just work. I believe this is true for a number of other Broadcom based switches, because the base is from OpenNSL and that is using Linux kernel 3.16.

root@OPX:/root# uname -a
Linux OPX 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u2 (2017-06-14) x86_64 GNU/Linux

GitHub - Broadcom-Switch/OpenNSL: Open Network Switch Library

 

[Jerry Renwick]

Nice to know about your experience, I am interest in Open Switch, too.

 

[AT S37=0]

I've been trying to determine if IGMP snooping works. The Linux bridge system properties for enabling it are set:

$ cat /sys/class/net/br100/bridge/multicast_snooping
1

If I tcpdump, I see multicast traffic on all interfaces, so I suspect that this is not working correctly or there is another setting somewhere.