32x40GbE $400 OBO if you are crazy

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Tiberizzle

New Member
Mar 23, 2017
25
11
3
124
That is in IB or ETH mode ? I assume ETH but not sure.
This switch supports Ethernet mode operation only. It does support RoCE (RDMA over Converged Ethernet), which is more or less Infiniband over Ethernet. Pretty much anything that supports DCB (datacenter bridging) or PFC (priority flow control) supports RDMA, though, and technically speaking neither is absolutely required for RoCE unless you're particularly attached to the data you're putting on the wire.

Does the ICOS SSH daemon also have a password and if so how to reset that
I assume it does. It's not the password you set in Ubuntu as far as I could tell. When I finally figured out there were two sshds and that the one my password worked for was on a different port I disabled the icos sshd without further investigation so I couldn't say for sure :p

Is that also required for basic 'switch' operations, where the switch is the single switch in the whole network, just connecting a bunch of switches, no up- and down-links, no bonding? Just plain "server with 40Gbe card connected to switch using a QSFP+ (DAC) cable?
It is not required in any circumstance where the default gateway for the management context is the same as the default gateway for the dataplane context, it is the only default gateway, or there is no default gateway in the dataplane context. I am not sure exactly what happens if multiple vrfs are configured, but my suspicion without consulting the documentation would be that gateways in vrfs other than the default dataplane vrf and default management vrf would not collide in this manner, and that routes in those vrfs would be installed in alternate kernel tables as well.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
If you know that the OS on the 4040 fills your needs then get it.
Me personally would stick with the 6036 (since you already got a 6012 and know mlx os), unless you are lacking something (while remembering that an official '36) can run the newest firmware unlike the converted '12))
 
  • Like
Reactions: Samir

psarossy

New Member
May 28, 2020
2
1
3
Sorry for the slight necroposting.

I picked up one of these Celestica Smallstone units, came with ICOS which seems to meet all I need, but all the documentation everywhere says what I'd expect, that you can use SR/LR optics or QSFP DAC/AOC cables. Based on this I picked up a bunch of mellanox nics and cables are they supposed to be compatible.

Looking at the switch I can't get the DACs to be recoginzed though as it says it only wants to do 40000baseLR4/Full and nothing else. I even got a test QSFP cable from FS.com since they sell the exact same switch. It detects the link, but says "Transceiver: external" and "Port: FIBRE" vs "Transceiver: internal" and "Port: Direct Attach Copper" I see on the NICs and I'd expect...

Can find any documentation (not did I find anything by digging in the config) on how to change it so it would be willing to use anything else than 40000baseLR4 (I guess 40000baseCR4)

Any hints from the folks who got it working?
 
  • Like
Reactions: Samir

psarossy

New Member
May 28, 2020
2
1
3
@RedX1 Yeah I bought one exactly like that, and now shipping it back, as even though every single spec says it should support a wide variety of transceiver modes, the one I got from eBay only is willing to accept 40000baseLR4 and I'm not buying LR4 optics for a 6ft link :p
 

Takrbark3

New Member
Dec 17, 2017
27
1
3
42
Hi!

Does this switch support:
- line-rate routing ( nat, vrf, wan: pppoe, ... ) ?
- "official" firmware are available ?

The only problem i found, it is an intel ATOM based CPU (just search: Atom C2000 bugs)
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
I need a 40GbE switch, 6610 with two ports is no longer enough. This looks like great value for the money, the only thing I need it for is running NVMe-oF. PFC would be nice, but RDMA seems to work really well without it too.
The other one I am considering is Arista DCS-7050QX-32S
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany

zhui

New Member
Oct 27, 2019
2
1
3
Okay, several points, and this is not meant as a personal attack against you and the wonderful work you do for the STH community here.

Running a brand name firmware (like modified IOS/JunOS/EOS) on OpenSwitch devices are a headache. Never mind the legality of grabbing name brand firmware, hacking it up to make it work (could be fun as an academic/hobbyist exercise, I used to do that in the Hackintosh/Hackintosh VM side of things), or the distribution of such an artifact, you are also depending on a single subject matter expert (SME), namely you, to hack-patch-release it. What happens if your work/personal life kicks in and free time goes to naught? Well, all those people who depended on you are SOL. When I buy hardware I don't plan to only keep it until the guy who release the software gets married and disappear - I prefer to buy something that doesn't have a single source for these kinda things, which is why I am glad i went with my Arista. Even if I go Cumulus it's still $799/year for access to software maintenance (which allows me to get the binary blobs that actually sets up the switch hardware, otherwise its just a large heavy noisy serial terminal), so mathwise, it doesn't make too much sense, even if the hardware is quite interesting. I might change my tune if Broadcom ICOS gets packaged into an RPM or pkg form, or if switchdev is a Broadcom hardware supported thing (it's only a Mellanox thing at the moment).

BTW, not everyone on the Enterprise side thinks whitebox switches are a great idea or are as promising as advertised. It is if you are Google/Facebook/Amazon scale, buys switches by the thousands, have a large devops team and have dedicated data center and remote hands staffing on the payroll 24/7. I don't have a large budget nor do I keep an inventory of spare parts, my staff can barely cope with the stuff we deal with already, and remote hands at my colocs are billable at $200/hour with 1 hour minimums. I would rather buy the name brand switch at the markups, get access to the long term support firmware versions, and then pay out the 27/4/365 support contracts so I can sleep better at night knowing that if something happens to my network equipment, it's not attributable to any failings on my part. My sales guys will also rip me a new one if the contract stipulates well known vendors and fully supported environments.

So no, I would rather snap up a Juniper or Arista on the cheap when they come on the secondary market (Would love a QFX5100 with that beefy Quadcore Ivy Xeon), and leave the whiteboxes alone for now.
Such a long post of ego BS. Why do you even mention your $799 subscription? Who cares.
 

zhui

New Member
Oct 27, 2019
2
1
3
Learn to comprehend English. Where did I say that I have a 799 dollar subscription?
Learn to comprehend English. Where did I say that I have a 799 dollar subscription?
"Your $799 subscription" sales pitch is what I mean. If you want to prompt it as a deal just post a separate post instead of attacking the most valuable contributor of this site and the servethehome community.
 
  • Like
Reactions: funkywizard

klui

Well-Known Member
Feb 3, 2019
824
453
63
Personally I don't think @fohdeesha took offense. @WANg has his points but I'm cheap and very much appreciate what @fohdeesha has done, regardless if he's the only person who could support these images. Beggars can't be choosers. Technology marches on and others will take the mantle when he decides to retire. But he's dropping lots of gold mines along the way https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-289896. You can't buy stuff like that. They are earned through constant discovery and kudos to him for sharing his findings from his adventures.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,729
3,083
113
33
fohdeesha.com
yeah I definitely didn't take offense, I can see where we're both coming from. Thank you for thinking I'm the most valuable contributor though :p (definitely not true however, I don't think I even ever leave the networking section). For anyone wondering, I *completely* lost interest in this project after getting a D4040 and realizing it has the cursed atom CPU - I didn't want to work on something that would inevitably brick itself, and for anyone else who bought one. It then went into the pile:




If anyone wants to work on it still for some reason, arista does publish their sources, and their version of coreboot was very easy to compile: Open-Source Software

that said Arista's have now come down so far in price on ebay I would just grab one of those - they're no longer $1300 versus $300, it's like $450 versus $300 now
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
My D4040 didn't come with rail guide posts on the back of my chassis. Does anyone know what kind of "screws" are they?

D4040 rail posts.jpg
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
"Your $799 subscription" sales pitch is what I mean. If you want to prompt it as a deal just post a separate post instead of attacking the most valuable contributor of this site and the servethehome community.
First of all, what "$799 subscription sales pitch"? I don't represent or work for Cumulus - I also mentioned ICOS and Switchdev as examples of free-as-in-beer alternatives.

Second of all, it's not an attack on @fohdeesha - I actually worked with him on the thread regarding the Arista 40GbE switches, looking at logs and adding (hopefully helpful) comments, and I have much respect for his work. More importantly I also have access to Arista EOS legally as part of my $dayjob.

Re-read what I am actually saying:

a) Factually speaking, @fohdeesha is currently the only person that I know of who modifies EOS for both Arista and non-Arista switchgear. if he drops off the radar (for various reasons, having a SigOth and needing to spend AFK time being one of them), well, that's it. And yes, someone would have to pick up his mantle - there are legal risks to what he is doing, and not everyone that has the technical knowledge would want to risk it.

That's because -

b) Running modified EOS on this switchgear is a grey area. EOS is only licensed to be used in Arista gear, and what you are getting from something like EOS is a modified Linux kernel+userland, and the binary blob and shims (kext/drivers) that talks to the switch ASICs - which is not open source and is something that Arista only validated for their equipment and paid for by their customers/subscribers. If you pay for Arista support and have legal access to EOS, great - you are still modifying their stuff to run it on non-Arista...which Arista will not be thrilled about. If you don't, you have even less to stand on.

It's not Linux itself that makes it legally problematic, it's the binary blobs and shims (ASIC drivers) bundled within that makes the switch work. Some of the open source distros include it (maybe an older version with blessing from Broadcom, Marvell or whichever ASIC maker you are dealing with), others require licensing (probably what you are paying for with something like Cumulus). It would be a great day if someone reverse engineer the ASICs and create a clean room, open source drivers for the switch ASICs, but it doesn’t exist, and until that day, you have to buy, borrow or steal. If you are not buying, do you consider yourself borrowing...or stealing?

It's legally the same as running Juniper JunOS or Checkpoint Gaia on something outside of their hardware, Cisco IOS on GNS3, MacOS on a Hackintosh - you are not supposed to do it (that's still true regardless of whether you are a current customer with legitimate access or not), but it's somewhat tolerated, or at least the lawyers for those companies didn’t consider it to be enough of an issue to go bother the hobbyists, but legally speaking, they are in the right and totally could put a stop to it - Apple went after Psystar and Cisco sent lawyers after people who unwisely bundled old IOS images on GNS3 and tried to sell them. Just because some end users here might be nonchalant about it doesn't make it a non-issue. If @fohdeesha gets lots of interest from hobbyists for modded EOS, that's great - but I am betting that a majority of interest came from people who already has legal access to the software in question but want the convenience of the mods that he made. But like I said, be aware of the legal ramifications - if the lawyers from Arista comes a-knocking, that distinction won't make much difference. I don't encourage others to do it, but ultimately, the choice is yours. I certainly don't think it's arrogance or ego for me to point this out, or whatever I am posting about alternatives it doesn't make it automatically a sales pitch.

c) This is a whitebox switch. Great for hobbyist use, but for enterprise use, unless you have an entire team behind you (and access to a legit distribution - something free like ONIL), it's not great. It's also from a vendor that turned its back on it (similar to the HPe EC200A saga, and partially why I threw my money behind an Arista instead). This is the 4th or 5th thread regarding the Celestica, and It's also not the first time someone mentioned the Atom C2xxx as being a ticking time-bomb (didn't Patrick make a video out of it less than 5 days ago?!), which should make you think twice.
 
Last edited:

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
yeah I definitely didn't take offense, I can see where we're both coming from. Thank you for thinking I'm the most valuable contributor though :p (definitely not true however, I don't think I even ever leave the networking section). For anyone wondering, I *completely* lost interest in this project after getting a D4040 and realizing it has the cursed atom CPU - I didn't want to work on something that would inevitably brick itself, and for anyone else who bought one. It then went into the pile:




If anyone wants to work on it still for some reason, arista does publish their sources, and their version of coreboot was very easy to compile: Open-Source Software

that said Arista's have now come down so far in price on ebay I would just grab one of those - they're no longer $1300 versus $300, it's like $450 versus $300 now
Hey man, you know that I always have your back when it comes to Arista and Brocade stuff, and you are legitimately a rockstar in here.
 
  • Like
Reactions: fohdeesha