32x40GbE $400 OBO if you are crazy

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Tiberizzle

New Member
Mar 23, 2017
25
11
3
124
Considering that there are 3 small diameter fans blowing between the 2 power supplies, I would say that it is likely VERY noisy.
Actually that's 5 (five) fans blowing between the 2 power supplies, which each also has a fan.

It's a high performance 1U datacenter switch and has the banshee wail you'd expect.

I haven't found a way to set the fan speed with the stock OS.
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
  • Like
Reactions: MiniKnight

MiniKnight

Well-Known Member
Mar 30, 2012
3,072
973
113
NYC
I always thought you needed like ONIE + x86 CPU + supported ASIC to get all this to work even if it's not supported.
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
The Celestica 8040 is x86 based (Avoton Atom C2558 if I remember correctly) with a Trident2 switch ASIC, so it's technically supported and probably worth a try.
 
Last edited:

Jason Hirsch

Member
Feb 12, 2018
36
6
8
47
So... as a guy that's getting out of the IT field (temporarily) because of ... leadership issues (theirs, which became mine obviously), would this be a good 'investment' to screw around with? Not that I have free time. However I'm not going to be traveling a ton, and 40gbE is exotic enough that there's a ton of stuff out there for cheap IF you pounce when it comes up.

-btw- both of ya'll in the firmware/enterprise discussion camp do great work. I'm a guy that loves digging into binaries, but I also switch the hat when I'm discussing hardware for enterprise. It's a fun ability to go back and forth and lets me set the BS detector off at appropriate levels. Unfortunately my leadership has a bit of 'turtle' mentality. Likes to 'trust the experts' (which means, not me). Didn't like if I asked if he only gets one quote on major house improvements. But that's as it goes.
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
So... as a guy that's getting out of the IT field (temporarily) because of ... leadership issues (theirs, which became mine obviously), would this be a good 'investment' to screw around with? Not that I have free time. However I'm not going to be traveling a ton, and 40gbE is exotic enough that there's a ton of stuff out there for cheap IF you pounce when it comes up.

-btw- both of ya'll in the firmware/enterprise discussion camp do great work. I'm a guy that loves digging into binaries, but I also switch the hat when I'm discussing hardware for enterprise. It's a fun ability to go back and forth and lets me set the BS detector off at appropriate levels. Unfortunately my leadership has a bit of 'turtle' mentality. Likes to 'trust the experts' (which means, not me). Didn't like if I asked if he only gets one quote on major house improvements. But that's as it goes.
Well, it's a fun little switch if you are trying to get into the open networking thing, but the obvious issue is the learning curve and applicability to the shops that you will end up in - it could be extremely good for learning purposes, or it could end up being one of those things that you'll have piling up in the corner. I bought an Arista switch at a very good price not too long ago (so I can give both SoNIC and EOS a try), but frankly, with the end-of-the-year projects @ work + home (need to get a MiniUSB power injection cable so I can run an external DVD burner on an old PowerBook G4, TV popped a fuse, wife's battery swap turned into a damaged loudspeaker, damn those Motorola Athene phones), I didn't even have a chance to take it home and really play with it yet.
 

jarekd

New Member
Apr 21, 2018
7
1
3
44
meh, EOS is just x86 fedora with a lot of custom shit on top making BRCM API calls to the ASIC. if it were a different ASIC I would say no, but both these switches are the exact same Trident II. Should just be a recompile of coreboot for the intel SoC board
Arista 7050SX-128 is Intel based

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Pentium(R) CPU @ 1.50GHz
stepping : 7
microcode : 0x29
cpu MHz : 1499.988
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes avx lahf_lm arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 2999.97
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
 

SPCRich

Active Member
Mar 16, 2017
256
137
43
42
Is there an easy/cheap way to get Cumulus Linux for ... evaluation/educational purposes?
Try reaching out to cumulus, they're stuff is licensed based, so they might be able to give you a trial key.
.
.
.
having said that, if you were able to somehow acquire CL, it DOES run without a valid license..because locking a switch up just because a license expired would be a bad idea I guess. IANAL/YMMV
 

jarekd

New Member
Apr 21, 2018
7
1
3
44
x86 = intel 8086 + successors (& clones) :D
It is obvious. But most of Aristas have AMD processors, not Intel. 7050SX-128 is small exception - not sure if aboot is the same for AMD and Intel based boxes.
 

40gorbust

New Member
Jan 12, 2019
25
2
3
Hi there, crazy checking in here I suppose. I purchased one of these a couple weeks ago and went ahead and purchased a second when I saw you lot found them :p
...
Hi,

I found the same switch on Ebay from a different vendor but he mentioned he doesn't have the Ubuntu password. Is it easy to hack/crack/start Ubuntu on the switch in single user mode like on a normal Linux PC?

I mean it won't be as easy as this How to Reset the Root Password in Linux - Make Tech Easier right ?

Seller's comments:

This sale is for 1 piece, although, there are 6 available.

These are recent pulls, fully functional. Each Switch has dual power supplies and 5 fans in the rear of the unit. Each unit has 32 QSFP ports in the front. These also have an EMC part number of 100-572-075-01

FYI, these units have a password and I do not know what it is. I do not know how to break or change the password.

The operating system that is loaded on these is Ubuntu OS version 14.04


Can you confirm that you can run near 40 Gbit ETH speed on the switch between two nodes/servers/clients connected to the switch ? What is your experience with this fine piece of Chinese manufacturing after a few months ?
 

Tiberizzle

New Member
Mar 23, 2017
25
11
3
124
Hi,

I found the same switch on Ebay from a different vendor but he mentioned he doesn't have the Ubuntu password. Is it easy to hack/crack/start Ubuntu on the switch in single user mode like on a normal Linux PC?

I mean it won't be as easy as this How to Reset the Root Password in Linux - Make Tech Easier right ?
It is trivial to reset the password, it boots as a standard Ubuntu system and from the serial console you can access BIOS, GRUB, and Linux command line. In order to reset the password you simply hit `e` at the GRUB screen to edit the default boot entry, throw an `init=/bin/bash` in there, wait a bit, and then `passwd`.

Can you confirm that you can run near 40 Gbit ETH speed on the switch between two nodes/servers/clients connected to the switch ? What is your experience with this fine piece of Chinese manufacturing after a few months ?
I have been able to sustain 37.7Gbps TCP/IP iperf3 via a single port of the D4040 in the following configuration:

Dual Xeon E5-2650V2 + Mellanox ConnectX3-EN -> D4040 -> D4040 -> Dual Xeon E5-2650V2 + Intel XL710-QDA2

This is around ~99.4% theoretical line rate with 78 byte TCP/IP, VLAN, and Ethernet framing overhead per 1542 bytes on the glass. Until I have time to run some more thorough tests I am willing to give it a 0.6% benefit of the doubt considering the system under test in that configuration was a virtual machine binding a single SR-IOV virtual function of the XL710-QDA2.

This result took a decent amount of tuning to achieve, but the ample documentation from both NIC vendors should point you in the right direction. In particular I found isolating the NIC's interrupts to specific CPUs in the virtual context and pinning those vCPUs to cores of the pCPU whose PCI bus the NICs were attached to be critical in sustaining >99% line rate, otherwise it dips by about 20-30% when cross-socket communication is involved.

Now, back to the switch. Some aspects of the switches default configuration are not what you would expect if you are familiar with pretty much any other vendor's hardware of this class. The device comes with two SSH daemons that can be configured independently: one that connects to ICOS directly, and one that connects to Ubuntu directly. I disabled the ICOS SSH and enabled the Ubuntu SSH.

You'll need to use Linux policy routing and multiple routing tables if you will learn/configure default routes for both the management and data plane interfaces (ip route .. table x, ip rule add ..., don't recall the exact config). Routes including the default learned from ICOS' routing protocols will be installed in the Linux main table, and so will routes configured for the management interface unless you specify an alternate table. There are separate interfaces for control plane to management interface, and control plane to data plane interfaces. They both must be configured if you want in band and out of band access to the control plane.

You have to specifically enable cut-through mode. There is no reason not to that I'm aware of and it is a significant reduction in latency for jumbo frame traffic. `cut-through mode` from configuration mode and a `reload` should do it. I do not believe cut-through will function between 40GbE and broken out 10GbE interfaces.

The default port bonding behavior (static bonding) does not work with anything that speaks LACP, and the default LACP configuration also will be difficult to bring up with anything that speaks LACP. In fact, while I'm here, I'll just save you the headache in the off chance you ever want to configure LACP on one of these.

The necessary and optional configuration for the lag interface:

interface lag 10
no port-channel static
port-channel load-balance 6
adminmode
The necessary and optional configuration for the lag member interfaces:

interface 0/1
port lacptimeout actor short
port lacptimeout partner short
no lacp actor admin state longtimeout
no lacp partner admin state longtimeout
The lag configuration sets the hash mode to L2+L3 and disables the static bonding mode, dynamic mode is LACP and that's what you more than likely want. The load balance mode is not strictly necessary, but if you are establishing a L3 adjacency traffic will of course only use one port of the bond in either direction without including some L3 header entropy in the hash. For lag interfaces `adminmode` is enable, `no adminmode` is disable. For everything else, it's `no shutdown` and `shutdown`. o_O The port configuration enables what's commonly referred to as active mode, and also sets short timeout (duh, right?). The lag will take much longer to come up (30s vs 1s) and to stop forwarding to partner lag members that become inactive without a link down event (90s vs 3s). In practice I found it would not come up at all with a Linux bond-driver partner unless I set active mode and would not reliably come up after a Linux bond-driver partner reboots until I set short timeout.

Anywho, I guess that concludes chapter 1 of my novel on the Celestica D4040 for now. Good luck and go to it ;)
 
  • Like
Reactions: Rand__

benbenben

New Member
Jan 19, 2019
3
0
1
Hi,

I found the same switch on Ebay from a different vendor but he mentioned he doesn't have the Ubuntu password. Is it easy to hack/crack/start Ubuntu on the switch in single user mode like on a normal Linux PC?

I mean it won't be as easy as this How to Reset the Root Password in Linux - Make Tech Easier right ?
It was for me. You'll need a console cable of course, but yes, it's just linux so resetting the root password isn't hard.

My question is, can I use hp infiniband qsfp+ cards with this? E.g.:

HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 661685-001 Low Profile 4872503154148 | eBay

Can I use these like a regular network card or are they only good for storage networking?
 

40gorbust

New Member
Jan 12, 2019
25
2
3
It was for me. You'll need a console cable of course, but yes, it's just linux so resetting the root password isn't hard.

My question is, can I use hp infiniband qsfp+ cards with this? E.g.:

HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 661685-001 Low Profile 4872503154148 | eBay

Can I use these like a regular network card or are they only good for storage networking?
https://forums.servethehome.com/ind...1-b21-mellanox-354-with-direct-connect.21632/

Looks like that will work. Good find, that's a very cheap card.
 

40gorbust

New Member
Jan 12, 2019
25
2
3
It is trivial to reset the password, it boots as a standard Ubuntu system and from the serial console you can access BIOS, GRUB, and Linux command line. In order to reset the password you simply hit `e` at the GRUB screen to edit the default boot entry, throw an `init=/bin/bash` in there, wait a bit, and then `passwd`.
Noice. We can do that. Have to dig up a serial cable and enable a port or get a USB to serial converter but that's no problem.

I have been able to sustain 37.7Gbps TCP/IP iperf3 via a single port of the D4040 in the following configuration:

Dual Xeon E5-2650V2 + Mellanox ConnectX3-EN -> D4040 -> D4040 -> Dual Xeon E5-2650V2 + Intel XL710-QDA2
That is in IB or ETH mode ? I assume ETH but not sure.

This is around ~99.4% theoretical line rate with 78 byte TCP/IP, VLAN, and Ethernet framing overhead per 1542 bytes on the glass. Until I have time to run some more thorough tests I am willing to give it a 0.6% benefit of the doubt considering the system under test in that configuration was a virtual machine binding a single SR-IOV virtual function of the XL710-QDA2.
99% speed is more than fine. Even >90% would be fine as it's a big improvement over 10 GBit ethernet.

This result took a decent amount of tuning to achieve, but the ample documentation from both NIC vendors should point you in the right direction. In particular I found isolating the NIC's interrupts to specific CPUs in the virtual context and pinning those vCPUs to cores of the pCPU whose PCI bus the NICs were attached to be critical in sustaining >99% line rate, otherwise it dips by about 20-30% when cross-socket communication is involved.
Indeed, before I found out you can 'assign' IRQs to cards we just kept moving the NIC around in the (dual-CPU) server and in slot 1 (close to CPU) it jumped from 20 Gbit to over 36/37 Gbit.

Now, back to the switch. Some aspects of the switches default configuration are not what you would expect if you are familiar with pretty much any other vendor's hardware of this class. The device comes with two SSH daemons that can be configured independently: one that connects to ICOS directly, and one that connects to Ubuntu directly. I disabled the ICOS SSH and enabled the Ubuntu SSH.
Does the ICOS SSH daemon also have a password and if so how to reset that ?

You'll need to use Linux policy routing and multiple routing tables if you will learn/configure default routes for both the management and data plane interfaces (ip route .. table x, ip rule add ..., don't recall the exact config). Routes including the default learned from ICOS' routing protocols will be installed in the Linux main table, and so will routes configured for the management interface unless you specify an alternate table. There are separate interfaces for control plane to management interface, and control plane to data plane interfaces. They both must be configured if you want in band and out of band access to the control plane.
Is that also required for basic 'switch' operations, where the switch is the single switch in the whole network, just connecting a bunch of switches, no up- and down-links, no bonding? Just plain "server with 40Gbe card connected to switch using a QSFP+ (DAC) cable?

You have to specifically enable cut-through mode. There is no reason not to that I'm aware of and it is a significant reduction in latency for jumbo frame traffic. `cut-through mode` from configuration mode and a `reload` should do it. I do not believe cut-through will function between 40GbE and broken out 10GbE interfaces.
I had to google cut-through haha, I guess I've been spoiled with plug 'n play 1Gbit and 10Gbit switches. I see it helps with latency but if latency is less of a problem and throughput is more important (e.g. for iSCSI/iSER and other (file)transfers) ; does it matter to have cut-through enable ?

Thanks for your extensive reply!
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
It was for me. You'll need a console cable of course, but yes, it's just linux so resetting the root password isn't hard.

My question is, can I use hp infiniband qsfp+ cards with this? E.g.:

HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 661685-001 Low Profile 4872503154148 | eBay

Can I use these like a regular network card or are they only good for storage networking?
just follow this https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015