Modding/upgrading Arista switches?

[oddball]

I've become sort of obsessed about Arista switches. I love that they're a Linux machine with a nice switch OS on top.

I have a 7050Q-16 that I took apart recently. It uses standard RAM and has a SATA slot on the motherboard.

Has anyone upgraded the RAM in these things?

Second has anyone tossed in an industrial flash, or SSD? Something like this (Industrial Disk on Module Kingspec SATA DOM 22Pins 128GB4CH Horizontal+Socket HS | eBay) ?

I found a doc floating around (https://www.arista.com/docs/Manuals/statement_of_volatility.pdf) that shows the internals for a number of models.

It looks like the 7124SX might be the most expandable of the platform. It includes a M.2 slot along with a SATA slot as well. The only issue with the 7124SX is the version of EOS it maxes out at. I believe in 4.17 or 4.18 they added the ability to run containers, which if you're modding these is very useful.

Interested in any experiences.

 

[Patrick]

I am not this brave, even with the cold spare sitting around.

 

[pcmoore]

I had written off Arista switches because I couldn't find an easy way to get easy access to official firmware updates without a support contract. Aside from the hardware tweaks you mentioned above, is it possible to replace the software stack on the device? If so, any pointers you can share?

 

[oddball]

So yes and no.. You can find EOS versions floating around. But for most of us who aren't buying current hardware there is a hard stop on what version is supported.

If you buy something like a 7050qx-32S you can install Azure's SONIC on it and get updates (it's open source and free).

But here's another way to look at this. Arista has built in support for guest VM's. Any OS in qemu format can run on the switch. And the switch very rarely hits the OS in forwarding mode. If you are using less than 100% of the ports it's almost impossible for switching to happen in software, not in silicon.

So you have a switch with a Linux hypervisor. You could even run Windows on these things. The biggest limitation is really the small storage and RAM that come out of the box.

 

[pcmoore]

Thanks for the info. I wasn't aware of the work Microsoft/Azure was doing with SONiC on real hardware; I knew they had a software switch, but I didn't realize it could run on proper switching hardware. The VM-on-a-switch is less interesting to me, but I realize that is a fairly unique capability that will appeal to a lot of people.

I might have to look into this if I ever find a deal on one of the supported switches. I just managed to score a Dell 8024 and 8132/N4032 pretty cheap this week, but it looks like neither are supported :/

 

[oddball]

Yeah, Sonic is interesting but it’s only supported on newer models.

I find the VM thing fascinating. We have two of these in production and I’m trying to snag a few to hack on. Specifically to run souped up specific purpose VM’s right on the switch.

My though is to a VM this is essentially a small server with a ton of NIC’s. I want to try running some IDS stuff right on the switch. Or alternatively this could be an interesting router/firewall device.

There are some really interesting applications with a VM and tap aggregation. The limiter in most of this is storage and RAM.

 

[pcmoore]

I wonder about the ability of the VM's CPU(s) to process the packet stream for more than a handful of ports at full line rate. While the switching fabric can obviously handle it, I have a hunch that the CPU's I/O interface can't handle a full load and I'm almost certain that the CPU itself couldn't process a full compliment of ports running at line rate. Perhaps Arista has some special sauce to make it possible, or at least filter traffic hitting the VMs.

I'm very curious to hear more about your experiments with packet processing in the VMs once you've had some time to play.

 

[oddball]

I wonder about the ability of the VM's CPU(s) to process the packet stream for more than a handful of ports at full line rate. While the switching fabric can obviously handle it, I have a hunch that the CPU's I/O interface can't handle a full load and I'm almost certain that the CPU itself couldn't process a full compliment of ports running at line rate. Perhaps Arista has some special sauce to make it possible, or at least filter traffic hitting the VMs.

I'm very curious to hear more about your experiments with packet processing in the VMs once you've had some time to play.

I found a paper discussing running the Bro IDS on an Arista switch. They had it setup where the VM read the events and offloaded to collectors on compute nodes. With a few nodes they were able to process 50Gbps real-time. They said the limitation to getting to 100Gbps wasn't the switch, but the collector nodes.

Now granted this isn't 1Tbps, or whatever these boxes can do, but it was impressive. The idea was to put an IDS like this on the ingress ports.

Another interesting application is running HAProxy right on the switch to do load balancing and fail-over.

I have a 7124SX arriving Thursday that I'll use to test this stuff. I am going to upgrade the flash to 16GB or 32GB, and toss in an SSD and more RAM. Goal is to get the thing fired up like that, then work on VM's.

In newer versions of EOS they natively support Docker. But since this is just Fedora I don't see why I can't install Docker myself and run containers on here.

Performance will be interesting. I'll keep the thread updated.

 

[oddball]

Looks like the 7124SX is a Raven board and the 7950qx-32S is a Crow.

Purchased a 16GB USB DOM flash chip and a weird 64GB SATA DOM industrial board.

It takes 10600e ram. Looking for two 4GB dimms to get up to 8Gb. Potentially more if can find them cheap.

 

[oddball]

Put the 16GB USBDOM in. It was a REALLY easy upgrade:

1) Copy EOS and boot-config and startup-config to a USB drive
2) Take out the old USB-DOM, pop in the new one
3) Boot, hit Ctrl-C and in Aboot copy over required files
4) Reboot

I now have a switch with 16GB of flash, where 15GB is available to be used.

I noticed boot time was a little faster too, probably due to flash speed. The old flash was 2GB and probably 5-6 years old at this point.

I purchased the flash from eBay, it was $28 for 16GB.

Next step is getting the SSD, it's on order from China. That's dicey. It's an industrial SSD on a small chip. It was $50 for 64GB. If it doesn't work I will just use a Samsung SSD with a traditional flat SATA connector and figure out how to mount the thing to the case.

Last step, and the one I'm most unsure of is the RAM.

Right now it has 2GB 1RX8 PC3-10600E-9-10-D0 ram. It appears to be unregistered. I tossed in some 10600R dims and the system attempted to boot then threw a zillion memory dumps. So the motherboard doesn't seem to support rdimm.

Will any unregistered 10600 work?

I need to figure out how to get pictures on the forum, I've been taking them.

 

[oddball]

An update on this.

Upgraded the ram in the 7124SX. Purchased DDR3-10600 ECC UDIMM's, they worked right out of the box.

The switch now has 8GB of RAM, 16GB of flash, and I have a 64GB industrial SSD coming in from China. I expect it to work out of the box as well.

I've had a lot of fun learning on this thing. I have a number of VM's installed. The VM's are a little wonky to get working. I have an older version of Debian and Fedora working fine. I tried to load Ubuntu/FreeBSD/Fedora (current) and they all hang. Looks like it's an issue with the KVM hypervisor. I need to figure out why the slightly older versions work, it's strange, it's a very cryptic message.

I also picked up a 7050QX-32S. This was a coup, found it on ebay for $950 BIN, done...

Opened it up, has the Crow board and a quad core CPU. CPU is a quad-core AMD G-Series GX-420CA with Radeon graphics. There is a single DIMM slot, appears to be the same as the others.

Plans for that are a cheap M.2 ssd card, and a 16GB DIMM. The processor is decent, with all cores you have performance similar to a E5-2620 v0, it's about half the speed of a E5-2609 single threaded, which for a switch is significant.

The 7050QX-32S has docker installed, I'm going to leverage that and toss on some monitoring images. The idea is to capture sFlow and host a monitor on the switch. I might toss Cacti on there in a VM as well.

One application that seems overlooked is running IDS' on these things. They bind their ethernet interface to either the management port or a vlan. I need to experiment with this, BUT my thought is if you had a DMZ vlan and an internal vlan you could bind an IDS collector to each on the same switch and route to the same processing node.

 

[oddball]

Not sure if anyone is actually following this, but I'm going to keep updating to chronicle things. If anyone else is interested in this it'll finally be available via Google.

More updates:
On the 7050QX-32S... it has an m.2 slot. So I did the most obvious thing in the world and ordered a WD Sata SSD from Amazon. The drive didn't come with a screw, but thankfully I have a giant bag of "computer screws" that I've saved over the years and I was able to secure the m.2 to the motherboard without an issue.

I didn't know what to expect when I booted into EOS. The BIOS detected the drive right away and EOS started throwing "EXT4 Not Found" errors. From Aboot I was able to 'ls /mnt/drive' and even copy things to it. I believe the drive came formatted with FAT32 out of the box, and the switch could see it. But FAT isn't what EOS is used to, and I was getting some weird "Bogus fat block size" on boot.

Here's what I did.

enable
bash
sudo fdisk /dev/sda

<follow prompts to create a new partition>
<write out changes>

sudo mkfs.ext4 /dev/sda1

exit
reload (save and confirm changes)

At this point the switch rebooted and I now have drive: as a mount point in the OS, I can also see /mnt/drive along with /mnt/flash. My /mnt/drive is 233GB (a 256GB SSD formatted down).

What's interesting is EOS dumped a few files on there, a drive.conf file, aquota.user, archive and a lost+found.

But it's fully functional and working nicely. So success!

I'm also looking at upgrading the RAM on this. The only issue is there's a single DIMM slot. They sell ECC UDIMM's in 8GB, and supposedly 16GB, but from what I've learned in talking to memory resellers the 16GB is a phantom that's impossible to source. If this had two DIMM slots then I could do 8+8 and get 16GB. Unfortunately I'm going to have to settle for 8GB, which should be ok.

Here's what's crazy to me about these switches. In the 7050QX-32S the chip has a Radeon GPU built in, meaning I should in theory be able to run a Windows VM on this thing. And given the AMD chip is as powerful as an i3 it might run respectably!

 

[oddball]

On the cpu chip, I have a cheap Lenovo laptop that I used for demos that is less powerful than what's in the Arista switch. I'm still trying to wrap my head around this. The laptop runs Win10 without an issue, which means this switch can too.

Here's a guide on getting Windows to run under *nix as a docker image. I'm going to give this a shot, because..... why not?

 

[oddball]

I'm way down the rabbit hole on this, but here is the graphics card that's part of the SOC: Radeon R5 Series Graphics Cards | AMD

Also note, based on this thread (Arista EOS Central – Arista 7060X CPU specs information) the 7060cx-32 has the same CPU.

 

[oddball]

Here's the link to Windows on Docker: Running Windows Server 2k16 with Docker under Linux, KVM or Virtualbox

 

[pcmoore]

Here's the link to Windows on Docker: Running Windows Server 2k16 with Docker under Linux, KVM or Virtualbox

It appears as though that is a guide for running Docker in a Windows VM which is running on a Linux host system via QEMU/KVM or VirtualBox (I suppose one could be running VirtualBox on a Windows host). Note quite the same as running Windows inside a Docker container on a Linux host system.

That said, this has been a very interesting thread!

 

[oddball]

@pcmoore good catch! Arista supports native qemu virtual machines, and from some googling it looks like you can just take a VMware image and get it to run.

I have a Windows 2012 VM laying around I might experiment with. The Windows thing is for fun, I want to see if it works.

The biggest limitation is the RAM. If I could get to 16GB or 32GB I'd do it without a second thought. But 8GB is limiting, my thought is to get very slim versions of these things running. So a VM/Container with 512Mb of ram. I don't want to drop below 2GB free, which I'll leave for the control-plane, so I will have about 6GB to work with.

But outside of that this is a great platform. I want to run a few VM's on here to monitor flows and network traffic. My original intent was monitoring in virtual machines, but I'm wondering if docker containers are easier to work with. Here's a great write-up (Containerizing my NSM stack — Docker, Suricata and ELK)

 

[BLinux]

i'm subscribing because this sounds interesting...

 

[oddball]

localhost#show version
Arista DCS-7050QX-32S-F
Hardware version:    01.10
Serial number:       JPE14170490
System MAC address:  001c.737b.3cff

Software image version: 4.20.5F
Architecture:           i386
Internal build version: 4.20.5F-8127914.4205F
Internal build ID:      311c887d-d677-4f60-97dd-54e4ef202868

Uptime:                 20 minutes
Total memory:           7661140 kB
Free memory:            6445860 kB

localhost#

localhost#dir drive:
Directory of drive:/
       -rw-        7168           Jun 14 20:11  aquota.user
       drwx        4096           Jun 14 20:12  archive
       drwx       16384            Jun 8 21:04  lost+found
       drwx        4096            Jun 8 21:09  var_archive.2018-06-08-21:15:01.dir
       drwx        4096            Jun 9 20:17  var_archive.2018-06-09-20:18:03.dir
       drwx        4096           Jun 12 19:05  var_archive.2018-06-12-19:15:02.dir
245998739456 bytes total (233127751680 bytes free)
localhost#

Put the 8GB UDIMM in and it worked out of the box. Machine now has 8GB and 256GB SSD, not bad with essentially an i3.

 

[oddball]

I need to pull the pictures off my phone and post here as well.