What's a good used 100G switch these days?

[VMman]

Hi All,

I would like some advice, I'm looking to setup a ceph storage cluster to learn with and I wanted to skip 40G and go for something more exotic like 100G

What is a good value switch that can still be used and has some life left?

I see some Edge-Core AS7712-32X listed on ebay for under $2k but I understand the earlier revisions are timebombs to the atom bug of the period.

If anyone has some insight on the subject, I would appreciate.

Thanks

 

[altmind]

are you looking for a switch for your HOME homelab?

 

[VMman]

I'm looking for something I can home lab while I learn and then move into the office for a small cluster.

 

[ano]

dx010 etc are all gone from ebay and all sites, for now, for home use I'd probbably sit on the fence, my ceph I'm considering just going 40G now :| kinda doh since nics are all 100G

 

[i386]

What does "good" mean to you?

For me it means free access (without a support contract) to firmware update like the switches from ruckus
or homelab friendly (third party transceivers support, fan speed settings via cli/ui whatever) like arista
or great "performance" stuff like mellanox.
I have ebay alarms setup for certain 100gbe switches from these vendors and so far the cheapest has been a ruckus switch with 4x qsfp28 ports for 4.2k $ (+ shipping and taxes) or 6k+ $ (+shipping and taxes) for 32x qsfp28 port arista switches...

There are some sub 1k$ switches on ebay but they have no firmware/nos available or possibly defect socs/cpus.

 

[NablaSquaredG]

I see some Edge-Core AS7712-32X listed on ebay for under $2k but I understand the earlier revisions are timebombs to the atom bug of the period.

You also need a NOS - And SoNIC sucks (in my opinion )

 

[VMman]

Thank everyone, I may have to rethink and pivot to something 25G with 100G uplinks rather, I don't want to end up with expensive paperweights
(I have enough)

 

[altmind]

mikrotik CRS518-16XS-2XQ. and you will learn a lot.

if you want to learn, it limits the brand selection significantly - cisco, hp/aruba and juniper(as knowledge may be only half-transferable).

 

[oneplane]

dx010 can still be had for less than $1k, but anything from Quanta/QCT would be fine too if it is using the same Broadcom sdk with the self-generating licenses...

 

[MistyMoon]

If you want to go cheap, maybe the Hohunet S7606X-2Z model of switch could be looked at? The chip used seems to be CTC7132 as far as I know. They both come from China.
Also I am very interested in other switch products from this manufacturer.

 

[Stephan]

If you build a cluster of three hosts, one dual port 100 Gbps NIC per server is sufficient for a meshed topology. Every server connects to every other using a dedicated link. Cheap and effective and no switch needed. Proxmox has documentation how to do this. Ceph will be a mixed bag because of the overhead and booby traps hiding in inconspicuous parameters like device write caching. Hard to saturate even 40/56 Gbps using cheap ConnectX3 on some benchmarks. At these speeds PCIe bandwidth is becoming an issue so two single port 100 Gbps cards could be better than one dual port.

 

[VMman]

If you build a cluster of three hosts, one dual port 100 Gbps NIC per server is sufficient for a meshed topology. Every server connects to every other using a dedicated link. Cheap and effective and no switch needed. Proxmox has documentation how to do this. Ceph will be a mixed bag because of the overhead and booby traps hiding in inconspicuous parameters like device write caching. Hard to saturate even 40/56 Gbps using cheap ConnectX3 on some benchmarks. At these speeds PCIe bandwidth is becoming an issue so two single port 100 Gbps cards could be better than one dual port.

I was considering a non-hyper hyperconverged deployment and wanted to also have the hypervisors connected to the storage.

I'm now considering a 25G switch with 100G uplinks like the Dell S5048F-ON. Using dual 25G nics on the HV's and 100G on the ceph nodes.

Is it worth chasing RDMA for Ceph using these high-speed switches??

 

[aosudh]

If you want to go cheap, maybe the Hohunet S7606X-2Z model of switch could be looked at? The chip used seems to be CTC7132 as far as I know. They both come from China.
Also I am very interested in other switch products from this manufacturer.

Yeah, those switches are well manufactured, I think
The size of it is just suitable for a tiny homelab. But the operating systems may be worse( They said it was based on ONL). Another point is that the asic of this Switch is only support eight 25 gigs of serdes, So it would only have one 100 gigabyte ports. And 6 40 gigabyte ports

 

[aosudh]

I was considering a non-hyper hyperconverged deployment and wanted to also have the hypervisors connected to the storage.

I'm now considering a 25G switch with 100G uplinks like the Dell S5048F-ON. Using dual 25G nics on the HV's and 100G on the ceph nodes.

Is it worth chasing RDMA for Ceph using these high-speed switches??

To be honest, there is no way to use up those HIGHEND speed without rdma.

 

[graczunia]

If you want to go cheap, maybe the Hohunet S7606X-2Z model of switch could be looked at? The chip used seems to be CTC7132 as far as I know. They both come from China.
Also I am very interested in other switch products from this manufacturer.

super interesting stuff. they appear to actually be the ODM for FS.com switches, here is for example the equivalent: S8550-6Q2C, 8-Port Ethernet L3 Switch, 6 x 40Gb QSFP+, with 2 x 100G QSFP28, Support MLAG - FS Germany

i guess the software will be the exact same then

 

[Iaroslav]

I have a similar request but for production infrastructure. We've been working with Aristas for a few years now - no issues.
It's hard now to find anything with >2 100G ports + 40G or even 10G, price is $5000+
The only switch that will minimally fit us is 7280SE-72, 48x10GbE (SFP+) & 2x100GbE MXP switch.
Doe's anybody know any alternatives?

 

[tinfoil3d]

Production-wise I don't think you can compare anything with Arista, Juniper or Cisco. And they're probably all gonna be over 5k.
Some people here say that Mikrotiks are okay for production but, well... Depending on your goals a recently reviewed CRS504-4XQ-IN may fit the purpose while being super cheap. But it's just 4 100gbe ports.

 

[Iaroslav]

Production-wise I don't think you can compare anything with Arista, Juniper or Cisco. And they're probably all gonna be over 5k.
Some people here say that Mikrotiks are okay for production but, well... Depending on your goals a recently reviewed CRS504-4XQ-IN may fit the purpose while being super cheap. But it's just 4 100gbe ports.

Mentioned Arista is almost ok, except for this weird MXP port, as far as I understand it may not take 100G-LR modules and just need a specific fiber cabling for different port speeds. I love Mikrotiks, but a bit afraid to set it on such a serious load and didn't even know about SFP modules compatibility to run in Juniper/Cisco environment.

 

[tinfoil3d]

@Iaroslav You should be looking at 7280SE-68 then but that still only has 2x100gbe...
MXP is interesting, it appears to be using same MPO-24 type connector? But it's likely MM not SM.

 

[tinfoil3d]

Yeah, it's their relabeled name for MPO24, for 7280 series it seems to be multimode fiber. https://www.arista.com/assets/data/pdf/Whitepapers/MXP_Ports_Technical_Guide.pdf