40 GbE Switches - Mellanox, FS.com, something else...

[Stril]

Hi!

I need some new switches for my small datacenter (2 buildings, each with about 20 servers).
At the moment, there 4 Dell N4000 switching are building the "core". Every server is connected to two of the Dell N4000 with an A/P bond. Additional, there are some Dell 55xx for the clients - also connected to the N4000s. Everything is redundant, but because of the RSTP, 50% of the links are "idle".

Now, I want to upgrade to 40/100GbE. I need some robust switches, but I do not need many features. I do not use any L3 features. Only:
- L2 basic switching
- RSTP
- MLAG
...perhaps ERPS in the future
- SNMP monitoring
- LLDP

What I found have been the following models below 10k each:

- Mellanox
The Mellanox SN2100 are looking great, but on the edge of my budget. There is very few expirience on them in internet forums, but Mellanox is running in datacenters for many years.

- Dell/Force10
The former F10-models are looking good, but I do not like their "trunk" config. If I want to configure a port to forward every VLAN, it blows up the whole config, because there is no config like "switchport trunk allowed vlan all".
Force10 had a great reputation, but I do not know, what happend after Dell integrated them.

- FS.com
FS.com does sell switches, now. I was never disappointed, when I bought fibre components, there, but I do not find ANY test or expirience on their switches, but they seem to support everything, I need and a lot more. The datasheets of the S5580 and S8050 are looking great and their price is good, but I am afraid to run in a lot of trouble with these products.
Did anybody here use these switches?

Can you give me a hint on how to find the right product?

Thank you
Stril

 

[i386]

What's your budget?
Do you need new switches? With support?
Or are used switches from ebay okay?

 

[Stril]

Hi!

I hope to get switches below 10k$, each (new without support). Perhaps, I will add some support.

Regards,
Stril

 

[Patrick]

What about using these? Arista DCS-7050QX-32 32x Port 40G QSFP+ Ethernet Switch Front to Rear | eBay

Not the best for sure, but pricing wise they cannot be beat.

 

[Stril]

Hi!

That is unbelievable cheap! Are these Arista models so unpopular, or what do you think, why they are that cheap?

 

[i386]

They are end of life, they don't get any feature updates anymore and arista will stop releasing security updates for these switches in summer 2018.

 

[Patrick]

They are end of life, they don't get any feature updates anymore and arista will stop releasing security updates for these switches in summer 2018.

100% correct.

There is a reason they are inexpensive. I even bought a spare since they were inexpensive and I figure there is a chance for failure.

 

[Stril]

...and I don't get the latest firmware without a subscription - right?

 

[Jerry Renwick]

I have used Mellanox ConnectX-3 and DAC cables in my lab—reliable and cheap solution, but I have never used their optical switches. As for the FS 40G switch, I am interested in S5850 48-port data center switch, good price, I might have a try.
It seems like a nice choice for TOR switching, plus their sales guarantee five year tech support.

 

[Stril]

It would be great, if you could share your expirience with the FS.com switches. but what i had to find out after asking twice: the "S-Series" does not support breakout-cables!

 

[Jerry Renwick]

To be honest, I don't know. I am planning to use the transceivers and cables.
I will consult with their tech support. Happy to tell you the latest news

 

[i386]

...and I don't get the latest firmware without a subscription - right?

No updates without a subscription

 

[Jerry Renwick]

If you buy S58 series switch with 10G and 40G slots, then they guarantee that you can use 40G to 10G breakout cables. If you buy the one with 100G ports, you can't use the 100G QSFP to 25G SFP28 DAC cables as there is no 25G port available. FYI, you can always ask their tech support to help you out.

 

[Stril]

FS.com wrote me, that ONLY the "N-Series" do support breakout-cables and of course, I could not find any hint on how to configure the breakout-ports on S-models in their CLI-reference.

The communication with FS.com is quite complicated....

 

[MiniKnight]

You'd be crazy to buy a FS.com first gen switch. ebay has sold more of those Arista switches than FS will sell of its first gen.

There's support if you find a bug but then there's the multi-billion dollar company who's had tons of switches deployed and already fixed bugs.

IF FS 32x40 was $3k then I'd consider it --- maybe --- for a test network.

If you're looking at FS may as well look at QCT. They're at least huge selling into the big hyperscale. Edgecore is one of the big hyperscale whitebox guys too and they've got 32x40 for under 7k.

 

[Blinky 42]

If you're looking at FS may as well look at QCT. They're at least huge selling into the big hyperscale. Edgecore is one of the big hyperscale whitebox guys too and they've got 32x40 for under 7k.

I would also say unless it is for a test setup and you have the $ to burn the FS.com switches seem pretty unknown quantities vs the Quanta units. You can get Quanta gear with their own cisco-wonky like CLI or white-boxed with Cumulus on it if you want to work on the SDN side of things more.

I inquired about the FS switches with my rep before they came out and after, and they didn't seem to have enough features we needed to justify the unknown quantity factor vs. price. Since they are using non-Broadcom silicon internally it isn't like you are just buying another knock-off arista/juniper/qct switch with known quantities with a different CLI and CPU on it for the packet processing.

 

[ServerSemi]

I was reading warranty policy on fs.com and noticed that all their switches come with a 1 year warranty. I'd rather buy used on ebay than buy a new one with just 1 year warranty when most other vendors offer +5 years or even lifetime warranties.

 

[AT S37=0]

ISince they are using non-Broadcom silicon internally it isn't like you are just buying another knock-off arista/juniper/qct switch with known quantities with a different CLI and CPU on it for the packet processing.

Ok, that would be interesting.. Ethernet Switch Catalog | FS.COM

The N series are intel + broadcom and use a broadcom icos base

The S5 series are freescale power and use a zebos base

Arista and Juniper do bring unique features because of the work they put into the OS, I'm not so sure that QCT is as unique. I'd hazard a guess that QCT and FS.com are similar OS-wise.

Update: Upcoming FS N Series Spine/Leaf Switch for Data Center | FS.COM

Compatible OS:
Cumulus Networks - Cumulus Linux
IP Infusion - OcNOS
Open Compute Project - Open Network Linux
Pica8 - PicOS

 

[Stril]

I think, you are right about FS.com switches...

The only thing, that stops me from buying the Arista 7050s is, that I did not find any source for the latest firmware. Do you have any hint on where I could find it?

Thank you!

 

[AT S37=0]

Best I can do is this:

Supported Devices and Platforms · Azure/SONiC Wiki · GitHub

I'm currently playing with this as time permits. It's another broadcom ONL platform.

In any case, you wouldn't run an OS that you found on a forum or the internets. That's like downloading windows 10 from the internet and running it. OS support is like extra toppings on your pizza, it ain't free.