Modding/upgrading Arista switches?

Discussion in 'Networking' started by oddball, May 18, 2018.

  1. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    137
    Likes Received:
    40
    #41
    spali and fohdeesha like this.
  2. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    137
    Likes Received:
    40
    The m.2 slot is directly to the right of the ASIC in the first picture.

    Hold off on the SATA SSD (non-m.2). I have two of these switches, and one has the slot the other doesn't. The one that doesn't was manufactured a year after the one that does. I don't know if they had a board revision or what. Make sure you have it before purchasing the crazy Chinese SATA SSD.
     
    #42
  3. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,173
    Likes Received:
    943
    Thanks for the pictures! that tells me a lot. At first I was dumbfounded at how different your 7050qx-32S crow board was from the picture I pulled from this thread - https://forums.servethehome.com/index.php?threads/upgrade-flash-storage-on-arista-7050qx-32.18113/

    Then I realized that thread was the non-S 7050qx-32 - THEN I realized it matches your raven management board exactly - so, it seems the original 7050qx-32 (non-S) shipped with a raven management card. interesting stuff

    I no longer see obvious SPI flash and SPI programming header on the crow board, I really hope they didn't stick the flash on the bottom of the PCB (really annoying when they do that). I guess I have some poking around to do when mine arrives

    How did you fasten down your M.2 ssd? It seems the board is missing the standoff that the m.2 SSD usually screws down into, hopefully it'll just take a standard size standoff

    Funny about the missing m.2 slot, the HP T620 PC using the exact same gx-420ca CPU had the same issue - half the ones off ebay had an m.2 slot, the other half (including mine) do not
     

    Attached Files:

    #43
  4. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    435
    Likes Received:
    157
    Hmmm. From what I have read so far about Arista, their mixing and matching of the management engines/the switch backplanes doesn't surprise me too much. If you look at the various varieties of the 7050 series (QX2, QX3, SX3, etc, etc, etc) it looks like they swap in/out parts to make the line-up.

    If you look at the Juniper side of things, that's how they do things as well. If you look at the Juniper QFX5100/5200 (hardware most similar to this Arista 7050), that has a Sandy or Ivy based CPU talking to a Broadcom switch ASIC. The SoC runs the kernel/userland for the OS, and then talks to the backplane ASIC using shared libs/kmods, similar to Arista. They also delineate JunOS model/featuresets by parsing the results from common UNIX commands like uname in the installer scripts. Of course, you cannot hack the Juniper hardware nearly as much as the Aristas (I don't remember seeing RAM slots on the QFX5100), and the fact that the Crows are using the GX420CAs aren't publicized by either Arista or AMD.

    BTW, I thought all t620 Plus have M.2 SATA slots, but only the early Revision A Jamestown models have populated mSATA headers.
     
    #44
    Last edited: Nov 14, 2018
  5. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    435
    Likes Received:
    157
    Hm. Has anyone try populating the machine with non-EEC (aka standard desktop) RAM to see if it boots up/works?

    I almost wish that Arista has an equivalent to the old and lamented JuniperClue Wiki that told you about the actual hardware found in the devices...that would be very useful.
     
    #45
  6. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    137
    Likes Received:
    40
    Yes, I tried normal ram, also ECC but not unregistered. You will either get the Arista... and it'll hang there forever. Or with some DIMM's it started to spit hex out at me.
     
    #46
  7. anlin

    anlin New Member

    Joined:
    Dec 8, 2016
    Messages:
    29
    Likes Received:
    6
    Really interesting stuff! I might have to look into Arista switches if (when) I decide to upgrade to 40GbE.


    Registered ECC RAM requires chipset/CPU support (usually only supported on higher end server oriented hardware) and doesn't make a ton of sense in a switch management board, so I'm not too surprised by the lack of registered memory support.
     
    #47
  8. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,173
    Likes Received:
    943
    switch got here, oddly the board is slightly different yet again! different CPU heatsink, and they left the regular SATA data & power connector slot unpopulated, so no SSD DOM for me. Thankfully it DOES have an M2 slot though, so I'll just order a 128GB m2 SSD

    https://i.imgur.com/QaPxEcX.jpg
     
    #48
  9. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    137
    Likes Received:
    40
    The satadom for booting goes on the connector on the left with a portion screwing into the white stud. You need that to boot.
     
    #49
  10. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    137
    Likes Received:
    40
    Or for now boot from USB.
     
    #50
  11. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,173
    Likes Received:
    943
    That's the connector for the usb Dom (what it uses to boot in its stock config). The sata Dom connector is the full size sata data and power header that's missing on mine

    [​IMG]
     
    #51
  12. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,173
    Likes Received:
    943
    pulled the management card out and they did indeed hide the SPI flash on the bottom of the PCB. after a lot of probing around though I did find a header that they broke the SPI pins out to for programming though, and figured out the pinout:

    [​IMG]

    Was able to then read and write arbitrarily to the SPI flash (so I no longer need to worry about bricking it, I can just write aboot back to the chip), did a complete dump of it which is linked below:

    [​IMG]

    Digging through the dump, the actual bootloader/BIOS that brings the CPU and hardware up is coreboot. coreboot then boots a small linux + busybox install, that's "aboot" that you actually see when you boot the switch (then aboot of course boots the full EOS linux environment). I'd imagine the coreboot serial input/output is on another debug header, once I find that, depending on how they compiled it, I should be able to configure coreboot to boot something stupid on this switch like windows 7. I really want to play half-life on it, need to find where they broke VGA output to, if anywhere. I have a feeling it's the header labelled DPM

    SPI flash boot chip contents: http://fohdeesha.com/data/other/bootloaders/arista-dumped.bin
     
    #52
  13. spali

    spali Member

    Joined:
    Nov 4, 2018
    Messages:
    32
    Likes Received:
    3
    Still waiting for mine... curious what surprise I will get ;)
    ROFL... we need to find the PCI lanes to add a decent GPU on it ;)
     
    #53
    fohdeesha likes this.
  14. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    137
    Likes Received:
    40
    Ah, this is really good info.

    There's a built-in GPU on the chip. My thought was if you ran Windows on KVM the graphics would pass through. You could RDP into the machine. I found some Windows KVM images I'm going to try at some point.

    I have the SATA DOM on one and not the other.

    For the m.2 ssd. You can screw in a small stud (sometimes included with the m.2 itself) and screw the m.2 ssd into that. That's what I did.

    Here are the Windows images: Windows Cloud Images - Cloudbase Solutions
     
    #54
    fohdeesha likes this.
  15. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,173
    Likes Received:
    943
    looks like there's at least this many variants of the crow mgmt board:

    [​IMG]

    also just FYI, the power monitoring in the output of show environment all / show environment power displays the output power from the PSU's in DC, it's not an accurate measurement of power being drawn by the switch. For instance, show env power on my switch reports 72w output, but my lab meter (+/-0.5% accuracy) shows 86 watts being consumed. Still very good for a 2tbps switch

    the "input current" in the output of show env power however is very accurate (agrees with my meter to 2 decimal places) and the switch has a near-unity power factor, so if you just multiply the displayed input current by your house voltage (generally 120) you'll get real power draw +/-3 or 4 watts
     
    #55
    Last edited: Nov 18, 2018
  16. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    435
    Likes Received:
    157
    Alright, so I got around to poking inside the switch, and more importantly, dancing inside the aboot environment. As @fohdeesha so graciously found out/discovered, yeah, it has a mini Linux environment called aboot with most binaries symlinked back to BusyBox, and it already contain quite a few juicy tidbits for the budding hacker. The switch is definitely quite noisy on the stand-up. I pulled 2 of the fans just for a bit more quiet since it'll not be doing much.

    So here's what I got so far poking around - I ordered a 32GB USB DOM slow-boated from China so it'll take a while to the point where I can install EOS...although I got a feeling I could've bought one of these and used a normal USB thmbdrive instead). I also received a unit without the SATA headers on the Crow SoC board.

    To connect to the serial port, it's your normal baby blue Cisco RJ45-to-DB9 console cable. I used a Prolific PL2303 with my MacBook Air so you'll need the drivers and CoolTerm. The console setting is 9600 8 N 1 on the USB serial port.

    Several utilities that you might find interesting on the aboot prompt:

    1) nvramtool -l mainboard
    This basically dumps the Coreboot parameter identifying the board type. In mine it reports the board as DB-FT3...which is an AMD Olive Hill devboard, and it seems to be well documented on Coreboot.org. So yes @fohdeesha it's possible to toss Win7 on the switch if you so desire.
    My guess is that you might be able to tweak the coreboot parameters to not require ECC on boot so you might be able to use Desktop RAM (1.35v only though, please. I only have 1.5v on hand in the office)

    2) burnMMX/burnP6/burnK7
    Turns out that the little switch has a couple of burn-in utils. When you get your switch delivered to the office you might want to slap it in a spare conference room and run this util for a bit, just to ensure hardware stability.

    3) Showtech
    That one shows you the SKU (product ID), the Serial Number, and the hardware revision.

    4) lspci -vvv
    Well, some interesting info -
    a) The eKabini Radeon 8400E VGA controller is 1002:9831 (but it doesn't resolve as so since it's not the prettyprinted version.
    b) The only PCIe x16 link seen is on the root complex (not sure if it has a special connection with the Radeon 8400E)
    c) Check out 3475:0001 - that's the device ID for the Arista switch SoC. That's connected via PCIe x1.
     
    #56
    Last edited: Nov 19, 2018
    fohdeesha likes this.
  17. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,173
    Likes Received:
    943
    if you have a regular usb flash drive you can run EOS now from the front USB port, just format it fat32 and drop an swi on it, then in aboot: umount /mnt/flash && mount /dev/sda1 /mnt/flash && boot flash:EOS-4.21.0F.swi
     
    #57
    cesmith9999 likes this.
  18. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    435
    Likes Received:
    157
    Yep. Got it running. That damned switch is LOUD on 100% fan speed. Any issues if I keep the fan on a more-or-less permanent 20% to save on my ears? Also, I am trying to get someone with a t620 plus to test their thin clients on 16GB DIMMs. I found an auction for the ECC unbuffered 16GB DIMMs but I'll need someone to see if they can get theirs to work (I need to verify that the eKabinis can handle the memory density)
     
    #58
    Last edited: Nov 19, 2018
  19. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,173
    Likes Received:
    943
    shouldn't be any issues just watch the output of "show environment all" for a couple days and make sure none of the temps approach the warning temp threshold. I've had mine on 30% with 4 ports active and it hasn't even broken half of the warning temp. Worst case "critical temp shutdown" is still enabled so if it does somehow get too hot it'll shut off instead of cooking itself. It has to get REALLY hot for that to happen though
     
    #59
  20. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,173
    Likes Received:
    943
    can everyone with 7050qx-32S models paste the output of "show version detail | grep -i Aboot-norcal"

    curious what the latest version in the wild is since they don't provide or package aboot separately
     
    #60
    Last edited: Nov 20, 2018

Share This Page