What's a good used 100G switch these days?

[Iaroslav]

Yeah, it's their relabeled name for MPO24, for 7280 series it seems to be multimode fiber. https://www.arista.com/assets/data/pdf/Whitepapers/MXP_Ports_Technical_Guide.pdf

Yes, you’re right. It’s barely enough 2x100G ports. But guess what? 7280SE-68 starts from 5k on eBay…

 

[tinfoil3d]

Yeah, that's a huge difference between two p/n, if i google it, i see some results for lower than 2k. But if you need it to hit 100g at few km then... and if it's for business purpose. And you'll need redundancy too, right?

 

[Iaroslav]

Yeah, that's a huge difference between two p/n, if i google it, i see some results for lower than 2k. But if you need it to hit 100g at few km then... and if it's for business purpose. And you'll need redundancy too, right?

That's what our new DC replied:
"Uplink cables between cabinet rows can be only over a LC singlemode cable per our datacenter.
If an another cable is required, it has to be reviewed by the datacenter and might no be accepted."
Yes, redundancy is a must, yet in 10 years we had zero problems with Arista switches or any other ones, except some old dell with rusty fans blocked.

 

[tinfoil3d]

Are the racks really far apart and they won't let you run your provided cable between them? It's a huge jump in price between the 68 and 72.

 

[oneplane]

Just put in some WiFi cards and call it good.

 

[Iaroslav]

Ok guys, forget about that weirdo, now I'm on some serious crossroads:
Juniper QFX5120-48Y-DC-AFO 48x25G + 8x100G
CISCO N9K-C93180YC-EX 48x25G + 8x100G
Arista DCS-7160-48YC6-R 48x25G SFP+ 6x100G
Any thoughts? Which one will keep you less stressed about any module compability issues and vendor-locks?

 

[vangoose]

Ok guys, forget about that weirdo, now I'm on some serious crossroads:
Juniper QFX5120-48Y-DC-AFO 48x25G + 8x100G
CISCO N9K-C93180YC-EX 48x25G + 8x100G
Arista DCS-7160-48YC6-R 48x25G SFP+ 6x100G
Any thoughts? Which one will keep you less stressed about any module compability issues and vendor-locks?

100Gb ports can usually be broken down to 4*25Gb or 2*50Gb or 40Gb/100Gb, you don't really need 48 physical ports and it's more flexible.
I have Arista DCS-7060CX-32S-F, it's great.

core>show interfaces status
Port       Name         Status       Vlan     Duplex Speed  Type         Flags Encapsulation
Et1/1                   connected    trunk    full   25G    100GBASE-CR4
Et1/2                   connected    trunk    full   25G    100GBASE-CR4
Et1/3                   connected    trunk    full   25G    100GBASE-CR4
Et1/4                   connected    trunk    full   25G    100GBASE-CR4
Et2/1                   notconnect   trunk    full   100G   Not Present
Et3/1                   notconnect   trunk    full   25G    100GBASE-CR4
Et3/2                   notconnect   trunk    full   25G    100GBASE-CR4
Et3/3                   notconnect   trunk    full   25G    100GBASE-CR4
Et3/4                   notconnect   trunk    full   25G    100GBASE-CR4
Et4/1                   notconnect   routed   full   100G   Not Present
Et5/1                   connected    trunk    full   25G    100GBASE-CR4
Et5/2                   connected    trunk    full   25G    100GBASE-CR4
Et5/3                   connected    trunk    full   25G    100GBASE-CR4
Et5/4                   connected    trunk    full   25G    100GBASE-CR4
Et6/1                   notconnect   routed   full   100G   Not Present
Et7/1                   notconnect   routed   full   100G   Not Present
Et8/1                   notconnect   routed   full   100G   Not Present
Et9/1                   connected    trunk    full   25G    100GBASE-CR4
Et9/2                   connected    trunk    full   25G    100GBASE-CR4
Et9/3                   connected    trunk    full   25G    100GBASE-CR4
Et9/4                   connected    trunk    full   25G    100GBASE-CR4
Et10/1                  notconnect   routed   full   100G   Not Present
Et11/1                  notconnect   routed   full   100G   Not Present
Et12/1                  notconnect   routed   full   100G   Not Present
Et13/1                  notconnect   routed   full   100G   Not Present
Et14/1                  notconnect   routed   full   100G   Not Present
Et15/1                  notconnect   routed   full   100G   Not Present
Et16/1                  notconnect   routed   full   100G   Not Present
Et17/1                  notconnect   trunk    full   100G   100GBASE-CR4
Et18/1                  notconnect   trunk    full   100G   Not Present
Et19/1                  notconnect   routed   full   100G   Not Present
Et20/1                  notconnect   routed   full   100G   Not Present
Et21/1                  notconnect   routed   full   100G   Not Present
Et22/1                  notconnect   routed   full   100G   Not Present
Et23/1                  notconnect   routed   full   100G   Not Present
Et24/1                  notconnect   routed   full   100G   Not Present
Et25/1                  notconnect   trunk    full   100G   100GBASE-CR4
Et26/1                  notconnect   routed   full   100G   Not Present
Et27/1                  notconnect   routed   full   100G   Not Present
Et28/1                  notconnect   routed   full   100G   Not Present
Et29/1                  notconnect   routed   full   100G   Not Present
Et30/1                  notconnect   routed   full   100G   Not Present
Et31/1                  notconnect   routed   full   100G   Not Present
Et32/1                  notconnect   routed   full   100G   Not Present
Et33                    connected    in Po1   full   10G    10GBASE-SR
Et34                    connected    in Po1   full   10G    10GBASE-SR
Ma1                     notconnect   routed   auto   auto   10/100/1000
Po1        ICX7250-LACP connected    trunk    full   20G    N/A

 

[Iaroslav]

100Gb ports can usually be broken down to 4*25Gb or 2*50Gb or 40Gb/100Gb, you don't really need 48 physical ports and it's more flexible.
I have Arista DCS-7060CX-32S-F, it's great.

Thank you! Price range is the same, but this one definitely beats them all, just need to find proper breakout cables/fiber for 25/10G links and that’s it!

 

[aosudh]

Thank you! Price range is the same, but this one definitely beats them all, just need to find proper breakout cables/fiber for 25/10G links and that’s it!

If you don't want to use those L3 characteristic , I think the plexxi PX-S3EQ-F can well satisfy your demand as it seems to be the OEM version of celestica DX010.

 

[tinfoil3d]

A whole different price range for those though I see one in Ukraine for ~6600 USD. Can't go wrong with that, but that's a huge jump.

 

[Iaroslav]

If you don't want to use those L3 characteristic , I think the plexxi PX-S3EQ-F can well satisfy your demand as it seems to be the OEM version of celestica DX010.

Thanks, but probably it's too exotic beast, XFP and too few of these

A whole different price range for those though I see one in Ukraine for ~6600 USD. Can't go wrong with that, but that's a huge jump.

Looks like just few REFs on local "market" right now. But we'll be ok to find it in EU as well. Years ago this Arista was a few times cheaper, how...

 

[i386]

Years ago this Arista was a few times cheaper, how...

Enterprises use the 100gbe stuff longer due to the lockdowns in the beginning of the pandemic and the following shortages, reducing/stopping the the supply for ebay or other platforms drastically.

 

[Iaroslav]

100Gb ports can usually be broken down to 4*25Gb or 2*50Gb or 40Gb/100Gb, you don't really need 48 physical ports and it's more flexible.
I have Arista DCS-7060CX-32S-F, it's great.

May I ask you if you have faced this issue? Let's say I want to link older X710-DA4 4xSFP+ card to this Arista, what's my best option?
Should I set it to 40G and may use:

Intel

Intel® Ethernet QSFP+ Breakout Cable

X4DACBL3

920347

606760003

QSFP+ to (4) SFP+ Direct Attach

Or it's not gonna work from switch side and I need:

Intel

Intel® Ethernet QSFP28 to SFP28 Twinaxial Breakout

XXV4DACBL2M

952303

NDAQGF-I202

100GbE/4x25GbE/4x10GbE Direct Attach

But set port as 4x25G operating in 4x10G mode, will it be accepted by NIC? As 100G breakout isn't listed in Intel® Product Compatibility Tool for X710-DA4, but for 2x25G XXV710-DA1/DA2

And what about single 40G XL710-QDA1 - will it work with regular QSFP twinax?

P.S. We're close to stick even with MikroTik due to it's baldness and three times lower price for a new one, as technically it's more than enough for our case, yet a bit afraid of newness and no positive feedback from Mtik course trainers

 

[tinfoil3d]

@Iaroslav just really be aware there's no RDMA in mt.

 

[oneplane]

Also keep in mind that mt tends to do weird topologies to get to that price point, so the management/fabric/serdes layout may be non-standard enough to have unexpected behaviour. (but still up to spec, but then again, who really constantly puts the spec and reality side-by-side all day long )

 

[aosudh]

@Iaroslav just really be aware there's no RDMA in mt.

I'm actually curious about whether mt's crs504 or 518 support RDMA as I have met some of the switches that were said to not support RDMA but they actually do support it.

 

[aosudh]

@Iaroslav just really be aware there's no RDMA in mt.

I have just read Japanese post about the 100Gbe network, a is local storage, acb and adb are passing through crs504 and 518, Proving that they all support rdma( The NIC are mcx623106 and E810

 

[i386]

There are different rdma over ether implementations (roce v1/v2, iwarp). They all will work even with the dumbest switch, but you want/need certain features (like dcb, pfc or others) to make sure that frames/packets are not dropped when there are problems in the network or when a port is maxed out...

 

[aosudh]

I have just read Japanese post about the 100Gbe network, a is local storage, acb and adb are passing through crs504 and 518, Proving that they all support rdma( The NIC are mcx623106 and E810

I didn't realize the differences in 4K.
Maybe mt should indeed make their products more standard and easy to use

 

[tinfoil3d]

I have just read Japanese post about the 100Gbe network, a is local storage, acb and adb are passing through crs504 and 518, Proving that they all support rdma( The NIC are mcx623106 and E810

Can you link the original post please?