Celestica Seastone DX010 32Port 100G QSFP28 $250

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
L

luckylinux

Active Member
Mar 18, 2012
791
211
43
  • Haha
Reactions: NablaSquaredG
I

i386

Well-Known Member
Mar 18, 2016
4,645
1,766
113
36
Germany
luckylinux said:
What would be the use case to install SONIC instead of OpenWRT on these, when used as a HomeLab Network Switch ?
Click to expand...
To get data from port A to port B (switching) :D

But seriously it's about having more up to date software compared to running a vendor os/firmware that is using outdated software (for example openssh server with known exploits), be it Sonic, Opewrt or other "nos" that might have slightly different goals/users. (The functionality/featureset is dictated by the used hardware and the asics do not change, so software won't get you more features)
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,854
1,234
113
There’s a reason why these switches are often comparatively cheap:

The lack of a good NOS.

SONiC is not a good NOS.

OpenWRT does not run on any big enterprise network switch (okay, maybe Mellanox, because these have mainline Linux kernel support), and is not a good NOS.


Oh and of course:
The ASIC is a Broadcom Tomahawk 1, which has lots of limitations.
And the switches have CPUs that are potentially affected by the AVR54 Atom C2000 bug.
 
BackupProphet

BackupProphet

Well-Known Member
Jul 2, 2014
1,289
860
113
Stavanger, Norway
intellistream.ai
Yeah, if you can wait, you will probably see good 100G switches below 500USD the next 12-18 months. In the meantime, just get either Mellanox SX6036 or Arista 7050QX-32(S). Both are awesome. Mellanox SX6036 also support 56G. Even if SSH is outdated, you do not run them on the internet. Preferably you have a seperate physical network to access these.
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,854
1,234
113
BackupProphet said:
good 100G switches below 500USD
Click to expand...
Depends what you define as „good“

Juniper QFX5200 (Tomahawk) are already around $500

Trident 3 switches below $500? Some models already are, but only Whitebox. Proper ones like Arista 7050X3 or Juniper QFX5120 below $500? We‘ll see.

Mellanox SN2000 is not quite there yet.
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,854
1,234
113
Same chip as 7060CX, DX010, Z9100, etc…

But good NOS (JunOS). Not quite as good as Arista, but definitely A-Tier.
 
L

luckylinux

Active Member
Mar 18, 2012
791
211
43
fohdeesha said:
yo dawg lovingly if you think openwrt is even a remotely viable OS to put on enterprise ASIC based 100g switches then you probably shouldn't be shopping for them
Click to expand...
Thank you for the "useful" Tip.

From where I stand, OpenWRT > any proprietary "OS" like Zyzel or HP(E) etc on the SMB 1gbps Series (e.g. Zyxel GS1900). At least you don't get Paperweights with extremely outdated and vulnerable Firmware.

Since I never used SONIC, I believe it's a legitimate Question, which still isn't answered ;).

So, what do you get, in exchange for no GUI and CLI-only ? I would assume the SOC Support / ASIC Network Switch (plus possibly the Management CPU/SOC, if separate, e.g. X86 on Mellanox SN2700 I believe) is baked into the Linux Kernel (plus some Firmware), so Device Support should be similar, since both are based on Linux.

Granted I have a liking to Debian, so SONIC could be interesting as well :) .

Exactly are the Pros vs Cons of SONIC vs something like OpenWRT ?
 
BackupProphet

BackupProphet

Well-Known Member
Jul 2, 2014
1,289
860
113
Stavanger, Norway
intellistream.ai
luckylinux said:
Granted I have a liking to Debian, so SONIC could be interesting as well :) .

Exactly are the Pros vs Cons of SONIC vs something like OpenWRT ?
Click to expand...
SONIC is maybe Debian under the hood. But you will quickly see how britle and fragile it is. As L2 switch it is ok, L3 can be hit and miss. I seems that OpenWRT works on Spectrum switches [OpenWrt Wiki] Mellanox Spectrum Switches but, you will lose important fucntionality. For example enabling ROCE is as easy as just writing: roce enable on MLNX-OS Onyx
 
  • Like
Reactions: luckylinux
L

luckylinux

Active Member
Mar 18, 2012
791
211
43
BackupProphet said:
SONIC is maybe Debian under the hood. But you will quickly see how britle and fragile it is. As L2 switch it is ok, L3 can be hit and miss. I seems that OpenWRT works on Spectrum switches [OpenWrt Wiki] Mellanox Spectrum Switches but, you will lose important fucntionality. For example enabling ROCE is as easy as just writing: roce enable on MLNX-OS Onyx
Click to expand...
I'm mainly interested in L2 Functionality (and Offloading) anyways.

I might be highly conservative but for me a Switch is Switching (L2), not Routing (L3). A Router is Routing (L3), not really intended to do Switching at Line Speed (L2).

All of these L2+ or L3 Light or L3 or whatever Switches (I assume mainly for Routing across VLANs) feels a bit "weird" to me, especially if you would like to have a Firewall in between your Subnets / VLANs (which you SHOULD definitively have).
 
S

seany

Member
Jul 14, 2021
31
25
18
luckylinux said:
I'm mainly interested in L2 Functionality (and Offloading) anyways.

I might be highly conservative but for me a Switch is Switching (L2), not Routing (L3). A Router is Routing (L3), not really intended to do Switching at Line Speed (L2).

All of these L2+ or L3 Light or L3 or whatever Switches (I assume mainly for Routing across VLANs) feels a bit "weird" to me, especially if you would like to have a Firewall in between your Subnets / VLANs (which you SHOULD definitively have).
Click to expand...
I think your missing a big use case for these kinds of switches: HCI

I have a similar model with 6 machines hooked up to it, but each machine has 1-4 k8s clusters on it and each one talks BGP to the switch. I have 10.0.0.0/16 divided into /24s routed through it. In my case I'm not doing any acl stuff, but simple segmentation would be ready to do. Dumping that out of the switch and into a "firewall" would be silly.
 
  • Like
Reactions: luckylinux
L

luckylinux

Active Member
Mar 18, 2012
791
211
43
seany said:
I think your missing a big use case for these kinds of switches: HCI

I have a similar model with 6 machines hooked up to it, but each machine has 1-4 k8s clusters on it and each one talks BGP to the switch. I have 10.0.0.0/16 divided into /24s routed through it. In my case I'm not doing any acl stuff, but simple segmentation would be ready to do. Dumping that out of the switch and into a "firewall" would be silly.
Click to expand...
I have zero Experience with that. I even had to look it up because the Acronym wasn't telling me anything :( .

Not sure why you have them talking BGP to the Switch. Couldn't you just do "normal" Routing with possibly Static Routes ?

I looked A BIT into BGP when I was considering getting some extra Public IP Addresses for my Homelab, but it didn't look very interesting/easy to setup (although in that case I needed a PUBLIC "Node" to make it work, like e.g. a VPS, Problem of course that you do NOT have in your Private IP Address Space). Not sure what would be the Advantages in such Case though ...
 
S

seany

Member
Jul 14, 2021
31
25
18
luckylinux said:
I have zero Experience with that. I even had to look it up because the Acronym wasn't telling me anything :( .

Not sure why you have them talking BGP to the Switch. Couldn't you just do "normal" Routing with possibly Static Routes ?
Click to expand...
Hyper converged infrastructure.


And I have no interest in managing 256 static routes, especially since they can change dynamically. BGP + rest apis, I haven't logged into that switch in years.


You just aren't the target market for those features it seems.
 
  • Like
  • Haha
Reactions: nedimzukic2, luckylinux and klui
K

klui

༺༻
Feb 3, 2019
997
585
93
NablaSquaredG said:
Oh and of course:
The ASIC is a Broadcom Tomahawk 1, which has lots of limitations.
And the switches have CPUs that are potentially affected by the AVR54 Atom C2000 bug.
Click to expand...
The only limitations of Tomahawk 1 I'm aware are it can't do L3 EVPN/VXLAN. Those in the know also say 25G port-to-port latency aren't as good as Tridents. Do you know of others?

Atom limitations does not apply to all switches that use the Tomahawk 1. For example, Arista 7060CXs use AMD GX-424CC, Juniper QFX5200s use Xeon E312xx. Celestica DX010, definitely buyer beware but safe if manufactured after 2018.
 
  • Like
Reactions: luckylinux
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,854
1,234
113
klui said:
The only limitations of Tomahawk 1 I'm aware are it can't do L3 EVPN/VXLAN
Click to expand...
It can, but only with recirculation. That’s just one of the limitations. Check the Arista Feature Matrix for 7050X(3) vs 7060X to get an impression

Tomahawk 1 has issues with FEC on breakout ports (no RS-FEC on 25G ports)

klui said:
Atom limitations does not apply to all switches that use the Tomahawk 1
Click to expand...
Sadly, most Tomahawk 1 switches have the Atom. Other examples are Dell Z9100 or Efge Core AS7712-32X
 
  • Like
Reactions: klui and luckylinux
You must log in or register to reply here.
Top Bottom