Switch licensing hell

dfector

New Member
Jul 19, 2021
17
0
1
Can anyone recommend a solid 48-port (1GbE) switch for top of rack duties?

I'm currently tinkering with a few bare metal Quanta LB9s, which seem like the perfect hardware for my project. But obtaining licenses is proving to be a nightmare. I've also looked into using Cumulus, but I can't justify the cost.

I'm only looking to do some basic L2 stuff (VLAN, LACP), no routing.

The Mikrotik CRS354-48G-4S+2Q+RM looks nice.

Are there other brands to look at that don't play these games with licensing and support contracts?
 

dfector

New Member
Jul 19, 2021
17
0
1
:D Touché!

I did see that Brocade thread. But I was wondering what the catch was. Are they not locked with a license key? And how difficult is it to obtain keys?
 

expressadmin

New Member
Jun 2, 2017
3
0
1
I actually have an LY9 that I am trying to revive from the dead.

The Quanta switches are really dependent on the CPU that is packaged with them. The "CPU board" that is packaged with the switch can range from a basic PPC P2020, to Atom (avoid due to AVR64 issues), and Rangeley CPUs (most desirable due to the flexibility that afford).

If you can find one with the Rangeley CPU, you can load a build of OpenNetworkLinux that is built for that platform/CPU combination. Here is a link that shows how to install ONL Quanta T3048-LY8/x86 (quanta-ly8-rangeley) Switch Reset Procedure
 

dfector

New Member
Jul 19, 2021
17
0
1
@expressadmin does that installation of ONL work for you? I keep reading posts about folks installing ONL, only to find that packet forwarding requires additional software. The documentation is a little sparse.

But I’m willing to give it a try. The ONL site lists the LB9 as a compatible switch.
 

RTM

Well-Known Member
Jan 26, 2014
763
279
63
I actually have an LY9 that I am trying to revive from the dead.

The Quanta switches are really dependent on the CPU that is packaged with them. The "CPU board" that is packaged with the switch can range from a basic PPC P2020, to Atom (avoid due to AVR64 issues), and Rangeley CPUs (most desirable due to the flexibility that afford).

If you can find one with the Rangeley CPU, you can load a build of OpenNetworkLinux that is built for that platform/CPU combination. Here is a link that shows how to install ONL Quanta T3048-LY8/x86 (quanta-ly8-rangeley) Switch Reset Procedure
Rangeley and Avoton are both codenames for Intel Atom CPU's in the C2xxx series and both may suffer from the AVR54 issue.
 

expressadmin

New Member
Jun 2, 2017
3
0
1
Rangeley and Avoton are both codenames for Intel Atom CPU's in the C2xxx series and both may suffer from the AVR54 issue.
Yeah, I have read the documentation and Intel has said that it may or may not be impacted by the same issue.

Honestly, it might be. My problem right now is that the switch can't see the internal mSATA storage. I am not sure if that is a bad device, or something related to the CPUs failure.

Regarding the ONL installation, see above... the mSATA device/port doesn't seem to work at the moment. I will let you know.
 

dfector

New Member
Jul 19, 2021
17
0
1
Is MikroTik a viable option? Or even Ubiquiti for that matter?

They both make 48 port switches (with additional SFP+ ports). And they both have perpetual software licenses included with the hardware.

The MikroTik even has redundant (albeit non-removable) PSUs and supports MLAG.

BUT... I assume there's a reason they're both < $400 new? :confused:
 

RTM

Well-Known Member
Jan 26, 2014
763
279
63
As you are not looking for L3 features, you could probably make do with switches from Mikrotik (they are working on L3 features) or Ubiquiti (some AFAIK supports L3 - or perhaps it is still WIP).

Mikrotik switches do not yet support MLAG in a stable firmware (it is only available in a recent dev release that can only be considered alpha quality), so I can not recommend buying for that feature.

The reason why they are cheaper, is that it is hard to call them enterprise quality switches.
I would put them as more intended towards SOHO/SMB/prosumer usage.
Ruckus/Brocade are actual Enterprise devices (atleast in my opinion).

At the end of the day, I think you need to figure out what you need.
You mentioned it is for TOR use, but only mainly 1G, so is it for your home lab? or are we dealing with something used in a more professional capacity?

If the latter, you probably want something a little more enterprise-y like a Brocade/Ruckus, however you may want to buy something new instead then (which may make a Mikrotik etc a better deal).

Anyway here are some more alternatives to Mikrotik/Ubiquiti that do not require a license to get updates and can be purchased for not too much money new:
  1. HPE Aruba 1930 (careful though, apparently it has issues with VLANs over LAG bonds)
  2. Cisco SG250X/SG350X and perhaps the new CBS350
My personal preference, would probably be a Ruckus ICX7150, they can usually be found cheaply enough on ebay, if you need the ports licensed legitimately look for the -4x10GR suffix in the model no. (or -2x10G if you can make do with 2 SFP+'s).
 

dfector

New Member
Jul 19, 2021
17
0
1
@RTM thank you so much for all the amazing insight!

You really saved me from another mistake. I almost bought four of the Mikrotik switches, but I didn't realize the MLAG feature wasn't in the stable firmware.

The switches will replace Cisco SG200s. They're used in a single rack colo environment.

The Quanta LB9 switches that I already purchased are perfect on paper. But I can't justify $1k Cumulus licenses; and Quanta hasn't given me a quote yet.

I'm definitely looking at the ICX7150 you recommended, as well as the ICX6610. I just want to be able to do MLAG with NIC teaming, to add some resiliency to our rack.

Thanks again for your help. Much appreciated.
 

expressadmin

New Member
Jun 2, 2017
3
0
1
@expressadmin does that installation of ONL work for you? I keep reading posts about folks installing ONL, only to find that packet forwarding requires additional software. The documentation is a little sparse.
So I finally got the switch alive. I did try the ONL image, but it didn't see any of the switch port interfaces. I am pretty sure I am missing something, I just don't know what it is.

I managed to find a slightly older copy of CumulusLinux (which the switch was originally running). We installed the license on CL and we are off and running.

It is a shame it is so loud otherwise I would run it in my home lab.
 

dfector

New Member
Jul 19, 2021
17
0
1
@expressadmin that’s cool that you got ONL running. After a little bit of reading, my understanding is that you have to install an additional package within ONL; and that software interacts with the switching ASICS. I saw mention of OF-DPA and OpenNSL as choices, but it sounded too involved for me to deal with.

I’ve also looked into Cumulus. But I can’t afford the $1k licenses… especially for EOL versions.
 

RTM

Well-Known Member
Jan 26, 2014
763
279
63
So we have established that MLAG is a "need to have" requirement. I am unsure if that is supported by the ICX7150.
My understanding is that it is generally a feature only/mostly found in high-end devices.
It would probably be a good idea to look into the Brocade megathread for answers for Brocade/Ruckus switches.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,276
2,356
113
30
fohdeesha.com
So we have established that MLAG is a "need to have" requirement. I am unsure if that is supported by the ICX7150.
My understanding is that it is generally a feature only/mostly found in high-end devices.
It would probably be a good idea to look into the Brocade megathread for answers for Brocade/Ruckus switches.
all the brocade models support stacking where they operate in an HA pair, so you could stack two ICX6610's for example, and LACP across both of them. Then you can lose a switch and still have connectivity, not sure if that satisfies your requirements
 
  • Like
Reactions: dswartz and RTM

dfector

New Member
Jul 19, 2021
17
0
1
@fohdeesha @RTM yeah, that HA feature is what I'm looking for. Different switch software calls it different things. QNOS uses the term MLAG. But it looks like FastIron calls it MCT.

So you're saying that all the Brocade models are capable of MCT, as long as the software is available?
 

dfector

New Member
Jul 19, 2021
17
0
1
Sadly, I have two LB9 switches set up exactly the way I want, using the latest QNOS software; but QCT refuses to sell me licenses. So in 60 days, it all stops working.

Does anyone have any "used" licenses they'd like to sell?

Thanks again to everyone here. I'm getting a real education on this stuff. :)
 

oneplane

New Member
Jul 23, 2021
15
7
3
Sadly, I have two LB9 switches set up exactly the way I want, using the latest QNOS software; but QCT refuses to sell me licenses. So in 60 days, it all stops working.

Does anyone have any "used" licenses they'd like to sell?

Thanks again to everyone here. I'm getting a real education on this stuff. :)
I find myself in the same boat here. Hardware is technically exactly what I am looking for (LB9) but QCT doesn't want to sell QNOS because it is out of support. My model seems to be previously used by Walmart Labs, has an old Cumulus Linux install with an expired license on it. Sadly it's an early revision with a single-core MPC8541 CPU and 1GB DDR1, but has working ONIE and working u-boot.

Part of me wants to reverse-engineer their licensing scheme and do things that makes the DMCA sad (we can't replicate actual keys because it's pubkey signed - but we can hack it or replace the public key), but it doesn't entirely sit right with me and I doubt a company like QCT would let that slide ;-)

I have a working QEMU setup with generic E500 PPC core support and a specific mpc8541 one so reverse-engineering and stepping through the code isn't all that hard.

Regardless, I've been looking at ONL and all it really needs is a way to get the ASIC to configure ports and TCAM via a remote controller (i.e. via a OpenVSwitch and open flow controller), which could run on a Pi or random x86 box. It's a bit much to get an older switch working but it has functional ports, can forward packets and has decent PPS/forwarding rates, and it would be a share to throw it away. They go for less than €90 on eBay including shipping, about 110USD.

Considering it doesn't have a whole lot of CPU power, porting SAI and SONiC to it with the heavy container workload is a bit unlikely to be useful. But a barebones CLI/SSH/REST/OpenFlow combo would fit. I'm currently looking at rebuilding ONL to see if it can be brought up-to-date.
 

dfector

New Member
Jul 19, 2021
17
0
1
Yeah. I’m not feeling all that bad about the idea of tinkering with the licensing mechanism. I’ve offered QCT my business, trying to do the right thing, and they’ve refused it. So I’m stuck with three very heavy doorstops.

Two of mine came in sealed boxes, one was from Walmart Labs (same, expired Cumulus license).

Sadly, this “experiment” all started by trying to avoid the ridiculous Cisco licensing headaches. Lesson learned.

I also looked into ONL. But I gave up when I realized that the switching hardware requires additional control software. Did you look at OF-DPA or OpenNSL? I think they run on hardware under ONL, but I’m not sure about ASIC compatibility.