Cheap switch 200/400GbE today equivalent of Celestica DX010 2 years ago

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
M

MichalPL

Active Member
Feb 10, 2019
189
25
28
Is there any light for cheap switches 200GbE or 400GbE on the ebay or other sources with used HW ?

What will be the future - better to switch from 100GbE into 200 or 400 ?

or... still using 2x 100GbE LACP ;) on ConnectX-4 PCIE 3.0 or ConnectX-5 PCIE 4.0?
 
M

MichalPL

Active Member
Feb 10, 2019
189
25
28
i386 said:
but it depends on the definition of "cheap" (and location)
Click to expand...
Yes cheap definition depends of the many things ;)

800GbE is too new


10GbE:
Reasonable Switch: about $350
10GbE card: $35
10GbE transceiver LC-LC: $5

40GbE:
Reasonable Switch: about $450
40GbE card: $44
40GbE transceiver LC-LC: $10

100GbE:
Reasonable Switch: about $500
100GbE card: $180
100GbE transceiver LC-LC: $8

and now 25GbE is similar to 100GbE - not worth to even try (comparing to the speed)

200GbE is astronomical (comparing to 100GbE, used), cost of the ConnectX-6 card is ~$700, transceiver and switch super expensive too.
is it clear what standard will be the cheapest for LC-LC fibers 200GbE?

400GbE even hard to find used components right now;)
 
  • Like
Reactions: Stephan
S

Sjhwilkes

New Member
Oct 17, 2020
28
2
3
vSphere 8 messes this up for me, my Mellanox ConnectX3Pro aren't supported, newer Mellanox or Intel 40 or 100g cards are very expensive. I may go 25g instead though it feels a waste of the 100g Celestica switches, I don't think my hosts will exceed 50g anyhow.
 
S

safado

Member
Aug 21, 2020
44
6
8
MichalPL said:
Yes cheap definition depends of the many things ;)

800GbE is too new


10GbE:
Reasonable Switch: about $350
10GbE card: $35
10GbE transceiver LC-LC: $5

40GbE:
Reasonable Switch: about $450
40GbE card: $44
40GbE transceiver LC-LC: $10

100GbE:
Reasonable Switch: about $500
100GbE card: $180
100GbE transceiver LC-LC: $8

and now 25GbE is similar to 100GbE - not worth to even try (comparing to the speed)

200GbE is astronomical (comparing to 100GbE, used), cost of the ConnectX-6 card is ~$700, transceiver and switch super expensive too.
is it clear what standard will be the cheapest for LC-LC fibers 200GbE?

400GbE even hard to find used components right now;)
Click to expand...
100gbe for 500? My brocade that I bought due to the awesome thread here is just about full on the 10gb ports. What’s out there that’s 25 or 100 that’s in the 500 price you mention? I must be out of the loop!!
 
C

Cruzader

Well-Known Member
Jan 1, 2021
548
557
93
Sjhwilkes said:
vSphere 8 messes this up for me, my Mellanox ConnectX3Pro aren't supported, newer Mellanox or Intel 40 or 100g cards are very expensive.
Click to expand...
The 40gbe cards above connectx3 are indeed fairly expensive.

ive replaced my hosts to keep the 40gbe switches "alive" a bit longer.
Was about 220/ea for cisco c240 m4 24sff CTO boxes + dual 40gbe mlom cards that are supported on 8, and got that back for the old units i replaced.
 
M

MichalPL

Active Member
Feb 10, 2019
189
25
28
i386 said:
Some whitebox switches that run certain builds of sonic or switches that might be affected by the intel atom problem.
Click to expand...
I think even cheaper bought 6 switches for about 380-460 usd, all working good (with fixed atom cpu)
Plus cards (required flash to work as 100GbE) for 100-120 EUR but long time ago, plus big transcievers box for about $8each.

Now maybe 400GbE will be possible somehow, and pcie 4.0/5.0 ;)

but… when switched computers from 1660v3 to 12900/12700/13900/13700 no enough pcie lanes for 100GbE
 
M

MichalPL

Active Member
Feb 10, 2019
189
25
28
Cruzader said:
The 40gbe cards above connectx3 are indeed fairly expensive.

ive replaced my hosts to keep the 40gbe switches "alive" a bit longer.
Was about 220/ea for cisco c240 m4 24sff CTO boxes + dual 40gbe mlom cards that are supported on 8, and got that back for the old units i replaced.
Click to expand...
I was using for a short while Cosco 3164Q (backthen cost about 1000 EUR) but switched to much faster (and cheaper) Celestica 100GbE -but less ports, and Mellanox 56GbE.

i am using:
ConnectX-3 for 40G
ConnectX-4 for 100G only
ConnectX-5 for 2x100G only
and having experimental 200G but no switches, no transcievers and waiting for 400G

so about $40 for 40G
 
S

Sjhwilkes

New Member
Oct 17, 2020
28
2
3
I'm a VCDX/vExpert so I get vSphere licensing for lab use for free. My hosts are Supermicro Twins, and I'm burning one of the two PCIe slots for an Intel Octane NVMe drive which is my vSAN cache layer, otherwise single port cards would be an option. I can see a handful of ConnectX4 for under 200, need to see if I can offload my NIB ConnectX3 56g cards to offset.
 
M

MichalPL

Active Member
Feb 10, 2019
189
25
28
Sjhwilkes said:
I can see a handful of ConnectX4 for under 200, need to see if I can offload my NIB ConnectX3 56g cards to offset.
Click to expand...
on ConnectX-3 working @ 40GbE i was able to saturate the link on samba under windows, 4.42GB/s.
I didnt try this @ 56GbE but should be faster.

I see right now pcie x8 for $100 and x16 for $159. (50GbE variant need to flash to 100GbE)
 
S

Sjhwilkes

New Member
Oct 17, 2020
28
2
3
So if I can get hold of CX414A PCIe X8 ConnectX4 there's a way to flash them from 2 X 50 to 2 X 100?
 
M

MichalPL

Active Member
Feb 10, 2019
189
25
28
Yes, but I don't recommend PCIe x8 for 2x100G, because exactly like Freebsd1976 wrote it will be about 80GbE (display as 100GbE link of course), what is not super bad for 40GbE replacement but it's not true 100GbE.

for 2x100GbE PCIe 3.0 x16 is also too slow (will do "160GbE") first good Mellanox card for 2x100GbE is ConnectX-5 PCIe 4.0 x16 - here you can have true 2x100GbE and should do 22.1GB/s file transfer when using LACP if NVMe and CPU will be fast enough.


sample commands under Win10 from my notes (for dual port card, single port is 455 or 415 - drivers from nvidia/mellanox page):

checking if card is visible and is based on mt4115 chip:
mst status

flint -d mt4115_pciconf0 -i c:\flash\mellanox\mt4155_2\fw-ConnectX4-rel-12_28_2006-MCX456A-ECA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin burn

or this one:

flint -d mt4115_pciconf0 -i c:\flash\mellanox\mt4155_2\fw-ConnectX4-rel-12_28_2006-MCX416A-CCA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin -allow_psid_change burn

(optional) fix ports to ETH only:
mlxconfig -d mt4115_pciconf0 set LINK_TYPE_P1=ETH
mlxconfig -d mt4115_pciconf0 set LINK_TYPE_P2=ETH
 
  • Like
Reactions: Stephan
You must log in or register to reply here.
Top Bottom