If you are looking at rack mounted stuff old mellanox sfp+/qsfp+ switches are fairly cheap.
I still got a sx1024 in use (48x sfp+ 12x qsfp+), those regularly appear on ebay in the 100-150$ area.
sx1016 with 64x sfp+ is maybe around 200$ and the popular sx6036 with 36x qsfp+ (and breakouts for your sfp+) is 30-100$ with plenty available.
sx1000 series and sx6000 are the same units, but 1000 is ethernet while 6000 is for infiniband with optional ethernet as added license.
(There are some threads about the sx6036 on here with information regarding enabling ethernet.)
I've seen some Arista or Celestica "reasonably" Priced switches at around 400-500 USD for QSFP28 (100gbe), but some have better support than others.
The Mellanox SX6036 could be a good Option for those of us that don't really need the Full Bandwidth and are on the Cheap. 40gbe realistic Target, cheap DAC & cheap Tranceivers. Personally I think QSFP28 Tranceivers are actually cheaper ... Look at this UK Seller, he has been selling lots of QSFP28 Transceivers at like 5 GBP / Piece, sometimes less. I don't think it's worth to expect / plan for 56gbps for a special Case of Transceivers, DACs, NICs, etc. Sure if you can get it for cheaper or same Price why not, but don't spend 10x more on it.
Question then becomes Transceivers/DAC Compatibility, because I read that QSFP28 Transceivers are NOT always compatible with QSFP+ Speeds ... unless they are plugged in a QSFP+ Port, which of course would apply to the Mellanox SX6036. Basically would a QSFP28 Transceiver work at 40gbps (is that Data Rate "supported") if plugged into a 40gbps QSFP+ Port ? I agree those QSFP28 CWDM4 Transceivers might be a bit "Special" (4 Light Wavelengths multiplexed into a single SMF Fiber), but for the point of view of the External Equipment, I guess that should be Transparent.
What about the SW Support of the Mellanox SX6036 though ? Running stock Mellanox Firmware or Flashed to SONIC / other ? Running "pure" Ethernet or "pure" Infiniband or "mixed" (1 Port Ethernet, 1 Port Infiniband) ? If running Infiniband I guess you need to run your own Subnet (?) Manager, right ?
Back then (1gbe-10gbe Ethernet vs 40gbps Infiniband) I recall Infiniband was definitively preferred for Block Storage and MUCH lower Latency, but I guess the Gap narrowed down when you compare 40gbe Ethernet / 40gbps Infiniband let alone 100gbe Ethernet / 100gbps Infiniband, so maybe we can get reasonably good Performance using much simpler Ethernet (TCP/IP).
My main Issue is probably going to be Noise though. There's a Reason why I never deployed my Voltaire Infiniband 4036 ... I tried to do a Fan Mod but I guess Priorities shifted around and ... 10 years later it's still not deployed with the Fan Module probably sitting somewhere I don't remember
.
Are these Mellanox SX6036 easy to water cool ? I never did Watercooling until now but that seems to be the Only Option. For the HEDGE EPYC we just discussed in the other Thread the Cooling Plate would cost twice as much as the Server so ... nope
. But maybe these Mellanox could work with a Water Block ? Unsure ...
EDIT 1: based on what I read in another Thread and IIRC, make sure that it's a REAL Mellanox SX6036 and NOT a Dell/etc branded one, otherwise I think the Ethernet License will NOT be easy to enable
.
