Switch licensing hell

oneplane

New Member
Jul 23, 2021
15
7
3
Yes, I did check out OF-DPA and OpenNSL, but neither has support for the switching ASIC in the LB9, it is a Firebolt3 which is in none of the public ASIC driver packages. The ones that do work are usually in the Trident, Helix, and Tomahawk series. ONL does work, but without the ability to control the switch ports it doesn't really help. I also wanted to contact Cumulus, but Nvidia bought them, and their contact page doesn't load (on the nvidia networking site).

You'd think that purely to prevent this functional (old/ancient) hardware landing in the trash they'd hand out legacy licenses for free at this point...
 

DavidRa

Infrastructure Architect
Aug 3, 2015
297
134
43
Central Coast of NSW
www.pdconsec.net
You'd think that purely to prevent this functional (old/ancient) hardware landing in the trash they'd hand out legacy licenses for free at this point...
Nope, there's no money in that. Apparently it's better business to fail to sell new switches to some than to sell licenses for old ones.

I'd also probably go for the secondhand ICX switches and just get a couple of spares. For the cost of the time to try to get licensing working, compared to having the business run ...
 

oneplane

New Member
Jul 23, 2021
15
7
3
I've quickly checked what a somewhat-minimal-branded reference design switch would go for, 48x 1GbE + 2x SFP+ from FS.com is about 600USD, with perpetual software license, same PPS and GbPS, and warranty... So essentially, getting a license if you can find a seller, combined with the cost of the second hand switch is about the same as a new one that has warranty and a longer lifespan.

I'd like it if someone here who does has a license file and matching MAC address is willing to share it with me, but I suppose it's a grey area...
 

oneplane

New Member
Jul 23, 2021
15
7
3
The one from FS probably just run ICOS, same as QNOS. Broadcom designs the hardware and a default software package, and 'brand' vendors then sell them with few or many customisations. The version of ICOS they ship depends on whatever they get from Broadcom.

I think ICOS, for example, comes with a lot of 'optional' things that have to be activated with individual licenses, QNOS, which is a branded ICOS mostly, only has 'do everything' licenses.

ICOS is some soft of FASTPATH successor, but looking under the hood, it all seems to be a singe big binary (switchdrv) doing all the heavy lifting. That same binary can be found on most non-OCP switches that are based on Broadcom designs. That process does the work on UniFi switches as well for example.

Due to the way they can select licensing strategies it doesn't seem to be as easy as just using any ICOS license anywhere.
 
  • Like
Reactions: dawsonkm

dfector

New Member
Jul 19, 2021
17
0
1
@oneplane wow! So is FastIron also a variant of ICOS? Would it be possible to run a FastIron binary made for a similar Brocade switch?

It would have to match on CPU, ASIC and bootloader?
 

oneplane

New Member
Jul 23, 2021
15
7
3
It would have to match the CPU, ASIC and Bootloader yes. That is also the problem we mostly have; one or more of those don't match up.

FastIron probably uses a monolithic application, it really depends on what SDK they got from Broadcom.

Generally there are a few components to it:

- Bootloader, this brings up the CPU and Memory and sometimes storage
- OS, this runs all applications
- Application(s), might be many smaller ones or one big fat 'do it all'

ICOS and QNOS generally do:

- U-boot as a loader
- Linux as the OS
- A few kernel modules and a switch application for the rest, and some extra applications for SSH, the CLI, and HTTP

Previously VxWorks was more popular, and that can do without U-boot on certain platforms.

- Some other loader as a loader
- VxWorks with embedded application, one big monolithic blob
- Maybe some storage with 'extra' applications, but that is not common with VxWorks

Modern NOS, ONL, SONiC etc, they do it differently

- Bootloader and/or Firmware, can even be PC-style BIOS or UEFI and GRUB instead of U-Boot
- Linux OS
- Perhaps a bunch of docker containers
- Abstraction interface which is the only place the 'magic' NDA software from the Switch ASIC vendor lives
- Common applications for switching, routing management

Many switches that try to deal with a market where Cisco is big try to mimic their CLI and with that some of the architecture. It's also faster and cheaper to just write all those 'embedded' things into one big do-it-all application. At the same time, it's less flexible, has a bigger risk (if that one big application crashes, the whole thing no longer works and has to reboot/reset), and for every switch hardware you have to recompile the entire thing. This is where SDKs at different levels come in. For the 'fat' applications they might use a 'do everything'-SDK, but for more modern approaches they might use a super smal ASIC-only SDK, or a somewhat bigger abstraction-SDK (i.e. with Linux Switchdev), or a complete SAI binary.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,333
2,475
113
31
fohdeesha.com
@oneplane wow! So is FastIron also a variant of ICOS? Would it be possible to run a FastIron binary made for a similar Brocade switch?

It would have to match on CPU, ASIC and bootloader?
fastiron (brocade software) is completely different from fastpath (broadcom software) - in fact the majority of brocade switching/routing products did not even run broadcom silicon until relatively recently (was all Marvell switching silicon)
 

oneplane

New Member
Jul 23, 2021
15
7
3
I've seen the same on some lower end Broadcom silicon switches where the SmartPath variant was used, they could nest it into their own RTOS, add it as a Linux payload or run it in VxWorks and the SDK would generate a firmware package all the same. A lot of basic HP switches used to be like that IIRC. Pretty odd to have u-boot, some no-name loader and VxWorks with their custom loader in the same product line, all starting different operating systems and application payloads, yet all from the same SDK for the same silicon line.
 

oneplane

New Member
Jul 23, 2021
15
7
3
A quick look at the FS.com switch OS firmware shows it is using a newer Broadcom SDK which does use very similar software (this time it's Linux on ARM), but it's newer and a bit more efficient. Some binaries are split out into more specific tasks and they are a lot smaller than the old monolithic switchdrvr. It also seems they aren't using any licensing mechanism, probably because they simply license all switches by default.

I imagine the licensing setup depends on how this works with Broadcom, perhaps they need to buy licenses per device from them, or they just buy an SDK version and then just build as many installs from that as they want. No clue how that is structured.
 

Serverking

The quieter you are, the more you can hear...
Jan 6, 2019
468
170
43
I’ve offered QCT my business, trying to do the right thing, and they’ve refused it.
Was about to put a $80k order in.... I guess I can no longer recommend them to my clients.
 

dfector

New Member
Jul 19, 2021
17
0
1
@Serverking yeah. I went in just trying to avoid Cisco (great products, nightmarish licensing).

The QCT switches were really well reviewed. And the “bare metal” aspect seemed appealing; I liked the idea of choosing an OS. Unfortunately, Cumulus is the only choice.

The stuff at fs.com looks nice. And the licensing looks pretty straightforward.
 

oneplane

New Member
Jul 23, 2021
15
7
3
Generally, the only BMS switches that are worth it are the ones that are supported by SONiC, SAI and/or OpenNSL. And for most of those, only if the CPU is x86 and in some cases ARM. PowerPC seems to be a dead platform for switches.

The Broadcom SDK still supports it, but as we can see from the firmware markings, those are considered 'legacy' now. A bit sad, since it still works physically, but then again, it is over 10 years old by now.

I think Quanta and others can still be good options, but it really depends on what you need. For a stand-alone switch, a BMS model isn't a good option at this time. But if you need multiple spine or leaf switches with a common controller, then getting something from the SONiC HCL is a good choice. I have been trying to get the 48x1GbE + 10Gb SFP model from Celestica but it seems to be nowhere to be found :(
 
  • Like
Reactions: Amrhn and klui

dfector

New Member
Jul 19, 2021
17
0
1
@oneplane so with Quanta, they sell the same switches under different part numbers? A BMS version and a QNOS version? The hardware is identical, just priced based on whether it has an OS?

It's strange that they don't have any kind of cross-licensing option, to take a BMS switch and load it with their QNOS software. :confused:
 

oneplane

New Member
Jul 23, 2021
15
7
3
@oneplane so with Quanta, they sell the same switches under different part numbers? A BMS version and a QNOS version? The hardware is identical, just priced based on whether it has an OS?

It's strange that they don't have any kind of cross-licensing option, to take a BMS switch and load it with their QNOS software. :confused:
Yep, that's essentially what they do. There are a few different hardware revisions per model, some earlier ones didn't have a Compact Flash card for example. That's also why the QNOS downloads show different ONIE installers for flash and CF.

One of the documents about the QNOS2 to QNOS upgrade (essentially they changed versioning schemes so QNOS2 is older than QNOS, there was some QNOS5 release in between but the newer versions have YEAR.MONTH and then an L suffix for legacy systems) shows a range of model numbers; there are about 6 different versions out there and about half of them are BMS (sold without QNOS license). It does somewhat make sense if you run your own software, why pay for a license that you don't need.

The downside is that Quanta is such a large company that there isn't really much of an 'in' to buy individual licenses. After prying a lot there does seem to be some regional reseller that apparently buys bulk licenses and then distributes them separately.
 
  • Like
Reactions: klui

dfector

New Member
Jul 19, 2021
17
0
1
@oneplane interesting. So do you know if the licensing is absolutely locked to a specific set of models, making it impossible for them to sell? Or is this just a matter of policy with Quanta?
 

oneplane

New Member
Jul 23, 2021
15
7
3
@oneplane interesting. So do you know if the licensing is absolutely locked to a specific set of models, making it impossible for them to sell? Or is this just a matter of policy with Quanta?
Licenses can be generated for any model, it's bound to the MAC address generally. So as long as someone is willing to take your MAC, put it in the license generator and send you back the resulting file it will work.

What I don't know exactly is how the pricing works. So the software mainly comes from Broadcom (ICOS Fastpath), and Quanta just gets that white label and brands it a little bit. Some pricing models say: every single license sale needs to be reported/paid to Broadcom. Others say: you have pay a contract which allows a fixed set of licenses to be given uit under your name. Yet others are simply: Quanta pays Broadcom once per firmware SDK version and can sell as many licenses as they want.

Say the SDK costs 1 million and you have about 3 million engineering cost to get your platform to work with the software (excluding hardware engineering cost etc), that means that you have to spend 4 million just to get everything in retail working order. That means that in order to make money you'd have to sell 4 million switches with 1 dollar markup to break even. Say they don't sell 4 million switches but only 50k, that means at least 80 dollars per switch to break even. Generally to do something retail you have to double that just to break even including supply chain, and then quadruple it before you can start making money after recouping R&D cost. So that brings the minimal expected price of a license around 320.

Say they already recouped all of their investment, paid Broadcom and their production lines can just churn out switches and your support team and software engineering team that does patches can support multiple models at once (thus sharing cost), then the cost to keep going goes drastically down. If they were to sell 10 different models but they all use the same software, you now only have 1/10th the cost of investment to keep that software running and then you can start making 'real' profit. I'm sure Quanta needs a lot of money to keep on going so this profit part probably only starts happening a few years after a product has come to market.

Most of this is just comparing actual hardware and software industries I've worked in with what I imagine Quanta might be doing so don't take this as a fact, more as an educated guess.
 
  • Like
Reactions: Amrhn and klui