Time to switch to 200GbE ? (QSFP56 and/or OSFP vs QSFP-DD)

[Michal_MTTJ]

I am evaluating options for upgrading to a 200GbE (from 100GbE) some small parts of the network environment and I am interested in understanding the realistic speeds achievable with different combinations of connectors and cables (never seen OSFP connector yet).

Probably QSFP56 is a good connector, or OSFP is better?

Now old used 200GbE QSFP56 Mellanox cards are similar price like 4 years ago Mellanox 100GbE QSFP28 (OSFP 200GbE and 400GbE are more expensive).

And now Switch - I found:

https://www.servethehome.com/dell-emc-s5248f-on-review-25gbe-to-200gbe-switch-ports/4/

and they have two 200GbE ports (enough for one NAS and one "C++ Builder"), but not QSFP56 but QSFP-DD 200GbE, any chance to connect it at 200GbE with Mellanox ConnectX-6 200GbE QSFP56?

I am guessing at 200GbE it might be tricky because QSFP56 is 4 lanes, QSFP-DD 200GbE is 8 lanes, so 100GbE easy 200GbE not possible or possible?

I am guessing that for example "Mellanox SN3700-VS2FO" will work perfectly 32 x 200GbE QSFP56 Ports, but $9000 for used one is still expensive (especially when max number of connected computers that ever need 200GbE is maybe 7 , right now maybe 3 make any sense).

Currently using Mellanox ConnectX-4 100GbE cards and 2x Celestica DX010, also ConnectX-3 40GbE connected to same Celestica @ 40GbE and few of them connected to Mellanox SX6036G switch then working in 56GbE mode instead of 40GbE.


What do you think - any ideas?
is QSFP56 better standard?
or OSFP 200 is better ?
or... even OSFP 400GbE - and than easy to connect QSFP56 cards @ 200GbE, easy to connect other cheap switches QSFP-DD @ 200GbE too and 400GbE for future?

Cards: OSFP or QSFP56, Transceivers (LC-LC) QSFP56 or OSFP ? or wait 3 years for 400

Best, Michal

 

[necr]

From my experience with QSFP-DD: https://forums.servethehome.com/ind...-already-running-at-200gbe.30563/#post-320310

I'd recommend to stay on QSFP56, it's really backwards compatible and the DACs are cheap. Next one is QSFP112, can't say that OSPF dominates the market - it's still a niche case. Transceivers: for anything small MTP/MPO, for a real installation something LC based with xWDM inside (single-mode).

Same problem as you, not too many links, test servers kinda old (PCIe 3.0/4.0), switches are hella expensive - I'll stay at QSFP28 in general, for some specific tests just connect NICs back-to-back or connect 2 servers to a server with a dual-port card.

 

[Michal_MTTJ]

Thanks! - Ok just bought first experimental (cheap used parts) QSFP56 setup to test it !

Dual port: I have two ConnectX-5 PCIe 4.0 2x 100GbE and my experience here is: CPUs are not fast enough to do LACP without RDMA.
Windows 10/11 SMB 3.0 + RDMA can achieve on any hardware 11GB/s (including $20 CPU's - E5 16xx v2), but without RDMA even 14900kf is too slow in single core to achieve more than 8-9GB/s realistically on ConnectX-5 (can't use RDMA and LACP at the same time - but maybe I am wrong here ?).

LACP is perfect to connect switches together.

Shortly will test Xeons W3495 and 8558P they have a lot of PCIe 5.0 lanes and I hope similar single core performance to 14700kf after small overclocking so maybe fast enough to saturate 2x 100GbE but I don't think so, almost sure it can saturate 1x 200GbE.

We have skipped MTP/MPO and RJ45 in the office and having only LC-LC (till now good decision for 10GbE/40GbE/100GbE, 56GbE only in rack).
Btw. at home also LC-LC (and minimal RJ45), 40GbE/56GbE cards and 100GbE switch - and ofc. 95% of the time using just WiFi

 

[Scott Laird]

FWIW, I don't think there's an easy way to connect a QSFP56 to a QSFP-DD at 200 Gbps today. One is 4 lanes of 50 Gbps, the other is 8 lanes of 25 Gbps. That pretty much rules DAC cables out, so you'd need optics for each that implement the same standard. fs.com only has 3 QSFP56 optics (200gbase-SR4, -LR4, and -FR4) and 2 QSFP-DD optics (both 200gbase-2SR4, which is 2x 100gbase-SR4, not 200gbase-SR4).

It looks like 200G LR4 QSFP-DD modules probably exist, but they're >$1k each and rare.

 

[Michal_MTTJ]

All initial parts received and the first (cheap) experimental 200GbE setup based on QSFP56 is working and looks promising for the future!

We have some fun in the morning "repairing" Mellanox 3700 (from ebay), after mechanical, electronic fix and one electric bypass it's finally working flashed Dell Mellanox ConnectX-6 100GbE card to OEM 200GbE, LC-LC QSFP56 still on the way but received copper 3meter 200GbE cables.