32x40GbE $400 OBO if you are crazy

Discussion in 'Great Deals' started by MiniKnight, Nov 25, 2018.

  1. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,851
    Likes Received:
    825
    ONIE means SONiC support right?
     
    #21
  2. Tiberizzle

    Tiberizzle New Member

    Joined:
    Mar 23, 2017
    Messages:
    22
    Likes Received:
    8
    Actually that's 5 (five) fans blowing between the 2 power supplies, which each also has a fan.

    It's a high performance 1U datacenter switch and has the banshee wail you'd expect.

    I haven't found a way to set the fan speed with the stock OS.
     
    #22
  3. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    431
    Likes Received:
    156
    #23
    MiniKnight likes this.
  4. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,851
    Likes Received:
    825
    I always thought you needed like ONIE + x86 CPU + supported ASIC to get all this to work even if it's not supported.
     
    #24
  5. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    431
    Likes Received:
    156
    The Celestica 8040 is x86 based (Avoton Atom C2558 if I remember correctly) with a Trident2 switch ASIC, so it's technically supported and probably worth a try.
     
    #25
    Last edited: Dec 9, 2018
  6. Jason Hirsch

    Jason Hirsch New Member

    Joined:
    Feb 12, 2018
    Messages:
    21
    Likes Received:
    4
    So... as a guy that's getting out of the IT field (temporarily) because of ... leadership issues (theirs, which became mine obviously), would this be a good 'investment' to screw around with? Not that I have free time. However I'm not going to be traveling a ton, and 40gbE is exotic enough that there's a ton of stuff out there for cheap IF you pounce when it comes up.

    -btw- both of ya'll in the firmware/enterprise discussion camp do great work. I'm a guy that loves digging into binaries, but I also switch the hat when I'm discussing hardware for enterprise. It's a fun ability to go back and forth and lets me set the BS detector off at appropriate levels. Unfortunately my leadership has a bit of 'turtle' mentality. Likes to 'trust the experts' (which means, not me). Didn't like if I asked if he only gets one quote on major house improvements. But that's as it goes.
     
    #26
  7. WANg

    WANg Active Member

    Joined:
    Jun 10, 2018
    Messages:
    431
    Likes Received:
    156
    Well, it's a fun little switch if you are trying to get into the open networking thing, but the obvious issue is the learning curve and applicability to the shops that you will end up in - it could be extremely good for learning purposes, or it could end up being one of those things that you'll have piling up in the corner. I bought an Arista switch at a very good price not too long ago (so I can give both SoNIC and EOS a try), but frankly, with the end-of-the-year projects @ work + home (need to get a MiniUSB power injection cable so I can run an external DVD burner on an old PowerBook G4, TV popped a fuse, wife's battery swap turned into a damaged loudspeaker, damn those Motorola Athene phones), I didn't even have a chance to take it home and really play with it yet.
     
    #27
  8. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,060
    Likes Received:
    429
    Is there an easy/cheap way to get Cumulus Linux for ... evaluation/educational purposes?
     
    #28
  9. jarekd

    jarekd New Member

    Joined:
    Apr 21, 2018
    Messages:
    4
    Likes Received:
    0
    Arista 7050SX-128 is Intel based

    processor : 0
    vendor_id : GenuineIntel
    cpu family : 6
    model : 42
    model name : Intel(R) Pentium(R) CPU @ 1.50GHz
    stepping : 7
    microcode : 0x29
    cpu MHz : 1499.988
    cache size : 3072 KB
    physical id : 0
    siblings : 4
    core id : 0
    cpu cores : 2
    apicid : 0
    initial apicid : 0
    fpu : yes
    fpu_exception : yes
    cpuid level : 13
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes avx lahf_lm arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips : 2999.97
    clflush size : 64
    cache_alignment : 64
    address sizes : 36 bits physical, 48 bits virtual
    power management:
     
    #29
  10. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,557
    Likes Received:
    367
    x86 = intel 8086 + successors (& clones) :D
     
    #30
  11. SPCRich

    SPCRich Member

    Joined:
    Mar 16, 2017
    Messages:
    83
    Likes Received:
    12
    Try reaching out to cumulus, they're stuff is licensed based, so they might be able to give you a trial key.
    .
    .
    .
    having said that, if you were able to somehow acquire CL, it DOES run without a valid license..because locking a switch up just because a license expired would be a bad idea I guess. IANAL/YMMV
     
    #31
  12. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,060
    Likes Received:
    429
    Excellent, thanks for the info/idea:)
     
    #32
  13. jarekd

    jarekd New Member

    Joined:
    Apr 21, 2018
    Messages:
    4
    Likes Received:
    0
    It is obvious. But most of Aristas have AMD processors, not Intel. 7050SX-128 is small exception - not sure if aboot is the same for AMD and Intel based boxes.
     
    #33
  14. necr

    necr New Member

    Joined:
    Dec 27, 2017
    Messages:
    14
    Likes Received:
    9
    Has anyone succeeded in running SONiC/Cumulus on these?
     
    #34
  15. 40gorbust

    40gorbust New Member

    Joined:
    Jan 12, 2019
    Messages:
    25
    Likes Received:
    1
    Hi,

    I found the same switch on Ebay from a different vendor but he mentioned he doesn't have the Ubuntu password. Is it easy to hack/crack/start Ubuntu on the switch in single user mode like on a normal Linux PC?

    I mean it won't be as easy as this How to Reset the Root Password in Linux - Make Tech Easier right ?

    Seller's comments:

    This sale is for 1 piece, although, there are 6 available.

    These are recent pulls, fully functional. Each Switch has dual power supplies and 5 fans in the rear of the unit. Each unit has 32 QSFP ports in the front. These also have an EMC part number of 100-572-075-01

    FYI, these units have a password and I do not know what it is. I do not know how to break or change the password.

    The operating system that is loaded on these is Ubuntu OS version 14.04


    Can you confirm that you can run near 40 Gbit ETH speed on the switch between two nodes/servers/clients connected to the switch ? What is your experience with this fine piece of Chinese manufacturing after a few months ?
     
    #35
  16. Tiberizzle

    Tiberizzle New Member

    Joined:
    Mar 23, 2017
    Messages:
    22
    Likes Received:
    8
    It is trivial to reset the password, it boots as a standard Ubuntu system and from the serial console you can access BIOS, GRUB, and Linux command line. In order to reset the password you simply hit `e` at the GRUB screen to edit the default boot entry, throw an `init=/bin/bash` in there, wait a bit, and then `passwd`.

    I have been able to sustain 37.7Gbps TCP/IP iperf3 via a single port of the D4040 in the following configuration:

    Dual Xeon E5-2650V2 + Mellanox ConnectX3-EN -> D4040 -> D4040 -> Dual Xeon E5-2650V2 + Intel XL710-QDA2

    This is around ~99.4% theoretical line rate with 78 byte TCP/IP, VLAN, and Ethernet framing overhead per 1542 bytes on the glass. Until I have time to run some more thorough tests I am willing to give it a 0.6% benefit of the doubt considering the system under test in that configuration was a virtual machine binding a single SR-IOV virtual function of the XL710-QDA2.

    This result took a decent amount of tuning to achieve, but the ample documentation from both NIC vendors should point you in the right direction. In particular I found isolating the NIC's interrupts to specific CPUs in the virtual context and pinning those vCPUs to cores of the pCPU whose PCI bus the NICs were attached to be critical in sustaining >99% line rate, otherwise it dips by about 20-30% when cross-socket communication is involved.

    Now, back to the switch. Some aspects of the switches default configuration are not what you would expect if you are familiar with pretty much any other vendor's hardware of this class. The device comes with two SSH daemons that can be configured independently: one that connects to ICOS directly, and one that connects to Ubuntu directly. I disabled the ICOS SSH and enabled the Ubuntu SSH.

    You'll need to use Linux policy routing and multiple routing tables if you will learn/configure default routes for both the management and data plane interfaces (ip route .. table x, ip rule add ..., don't recall the exact config). Routes including the default learned from ICOS' routing protocols will be installed in the Linux main table, and so will routes configured for the management interface unless you specify an alternate table. There are separate interfaces for control plane to management interface, and control plane to data plane interfaces. They both must be configured if you want in band and out of band access to the control plane.

    You have to specifically enable cut-through mode. There is no reason not to that I'm aware of and it is a significant reduction in latency for jumbo frame traffic. `cut-through mode` from configuration mode and a `reload` should do it. I do not believe cut-through will function between 40GbE and broken out 10GbE interfaces.

    The default port bonding behavior (static bonding) does not work with anything that speaks LACP, and the default LACP configuration also will be difficult to bring up with anything that speaks LACP. In fact, while I'm here, I'll just save you the headache in the off chance you ever want to configure LACP on one of these.

    The necessary and optional configuration for the lag interface:

    The necessary and optional configuration for the lag member interfaces:

    The lag configuration sets the hash mode to L2+L3 and disables the static bonding mode, dynamic mode is LACP and that's what you more than likely want. The load balance mode is not strictly necessary, but if you are establishing a L3 adjacency traffic will of course only use one port of the bond in either direction without including some L3 header entropy in the hash. For lag interfaces `adminmode` is enable, `no adminmode` is disable. For everything else, it's `no shutdown` and `shutdown`. o_O The port configuration enables what's commonly referred to as active mode, and also sets short timeout (duh, right?). The lag will take much longer to come up (30s vs 1s) and to stop forwarding to partner lag members that become inactive without a link down event (90s vs 3s). In practice I found it would not come up at all with a Linux bond-driver partner unless I set active mode and would not reliably come up after a Linux bond-driver partner reboots until I set short timeout.

    Anywho, I guess that concludes chapter 1 of my novel on the Celestica D4040 for now. Good luck and go to it ;)
     
    #36
    Rand__ likes this.
  17. benbenben

    benbenben New Member

    Joined:
    Jan 19, 2019
    Messages:
    1
    Likes Received:
    0
    It was for me. You'll need a console cable of course, but yes, it's just linux so resetting the root password isn't hard.

    My question is, can I use hp infiniband qsfp+ cards with this? E.g.:

    HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 661685-001 Low Profile 4872503154148 | eBay

    Can I use these like a regular network card or are they only good for storage networking?
     
    #37
  18. 40gorbust

    40gorbust New Member

    Joined:
    Jan 12, 2019
    Messages:
    25
    Likes Received:
    1
    https://forums.servethehome.com/ind...1-b21-mellanox-354-with-direct-connect.21632/

    Looks like that will work. Good find, that's a very cheap card.
     
    #38
  19. 40gorbust

    40gorbust New Member

    Joined:
    Jan 12, 2019
    Messages:
    25
    Likes Received:
    1
    Noice. We can do that. Have to dig up a serial cable and enable a port or get a USB to serial converter but that's no problem.

    That is in IB or ETH mode ? I assume ETH but not sure.

    99% speed is more than fine. Even >90% would be fine as it's a big improvement over 10 GBit ethernet.

    Indeed, before I found out you can 'assign' IRQs to cards we just kept moving the NIC around in the (dual-CPU) server and in slot 1 (close to CPU) it jumped from 20 Gbit to over 36/37 Gbit.

    Does the ICOS SSH daemon also have a password and if so how to reset that ?

    Is that also required for basic 'switch' operations, where the switch is the single switch in the whole network, just connecting a bunch of switches, no up- and down-links, no bonding? Just plain "server with 40Gbe card connected to switch using a QSFP+ (DAC) cable?

    I had to google cut-through haha, I guess I've been spoiled with plug 'n play 1Gbit and 10Gbit switches. I see it helps with latency but if latency is less of a problem and throughput is more important (e.g. for iSCSI/iSER and other (file)transfers) ; does it matter to have cut-through enable ?

    Thanks for your extensive reply!
     
    #39
  20. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,108
    Likes Received:
    854
    just follow this https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015
     
    #40
Similar Threads: 32x40GbE $400
Forum Title Date
Great Deals Foritgate 50E, 60E, 80E for $243, $210, $400 in CDW Outlet Friday at 2:06 PM
Great Deals EVGA GTX 1080 for $400 Aug 3, 2018
Great Deals AMD EPYC 7351P - $400 BO May 12, 2018
Great Deals New 1.92TB SanDisk Cloudspeed Eco Gen2 $400 Mar 26, 2018
Great Deals Supermicro 4U 846 with A backplane & 920W SQ PSUs - $400 Jul 25, 2017

Share This Page