Time to switch to 200GbE ? (QSFP56 and/or OSFP vs QSFP-DD)

[Michal_MTTJ]

I am evaluating options for upgrading to a 200GbE (from 100GbE) some small parts of the network environment and I am interested in understanding the realistic speeds achievable with different combinations of connectors and cables (never seen OSFP connector yet).

Probably QSFP56 is a good connector, or OSFP is better?

Now old used 200GbE QSFP56 Mellanox cards are similar price like 4 years ago Mellanox 100GbE QSFP28 (OSFP 200GbE and 400GbE are more expensive).

And now Switch - I found:

https://www.servethehome.com/dell-emc-s5248f-on-review-25gbe-to-200gbe-switch-ports/4/

and they have two 200GbE ports (enough for one NAS and one "C++ Builder"), but not QSFP56 but QSFP-DD 200GbE, any chance to connect it at 200GbE with Mellanox ConnectX-6 200GbE QSFP56?

I am guessing at 200GbE it might be tricky because QSFP56 is 4 lanes, QSFP-DD 200GbE is 8 lanes, so 100GbE easy 200GbE not possible or possible?

I am guessing that for example "Mellanox SN3700-VS2FO" will work perfectly 32 x 200GbE QSFP56 Ports, but $9000 for used one is still expensive (especially when max number of connected computers that ever need 200GbE is maybe 7 , right now maybe 3 make any sense).

Currently using Mellanox ConnectX-4 100GbE cards and 2x Celestica DX010, also ConnectX-3 40GbE connected to same Celestica @ 40GbE and few of them connected to Mellanox SX6036G switch then working in 56GbE mode instead of 40GbE.


What do you think - any ideas?
is QSFP56 better standard?
or OSFP 200 is better ?
or... even OSFP 400GbE - and than easy to connect QSFP56 cards @ 200GbE, easy to connect other cheap switches QSFP-DD @ 200GbE too and 400GbE for future?

Cards: OSFP or QSFP56, Transceivers (LC-LC) QSFP56 or OSFP ? or wait 3 years for 400

Best, Michal

 

[necr]

From my experience with QSFP-DD: https://forums.servethehome.com/ind...-already-running-at-200gbe.30563/#post-320310

I'd recommend to stay on QSFP56, it's really backwards compatible and the DACs are cheap. Next one is QSFP112, can't say that OSPF dominates the market - it's still a niche case. Transceivers: for anything small MTP/MPO, for a real installation something LC based with xWDM inside (single-mode).

Same problem as you, not too many links, test servers kinda old (PCIe 3.0/4.0), switches are hella expensive - I'll stay at QSFP28 in general, for some specific tests just connect NICs back-to-back or connect 2 servers to a server with a dual-port card.

 

[Michal_MTTJ]

Thanks! - Ok just bought first experimental (cheap used parts) QSFP56 setup to test it !

Dual port: I have two ConnectX-5 PCIe 4.0 2x 100GbE and my experience here is: CPUs are not fast enough to do LACP without RDMA.
Windows 10/11 SMB 3.0 + RDMA can achieve on any hardware 11GB/s (including $20 CPU's - E5 16xx v2), but without RDMA even 14900kf is too slow in single core to achieve more than 8-9GB/s realistically on ConnectX-5 (can't use RDMA and LACP at the same time - but maybe I am wrong here ?).

LACP is perfect to connect switches together.

Shortly will test Xeons W3495 and 8558P they have a lot of PCIe 5.0 lanes and I hope similar single core performance to 14700kf after small overclocking so maybe fast enough to saturate 2x 100GbE but I don't think so, almost sure it can saturate 1x 200GbE.

We have skipped MTP/MPO and RJ45 in the office and having only LC-LC (till now good decision for 10GbE/40GbE/100GbE, 56GbE only in rack).
Btw. at home also LC-LC (and minimal RJ45), 40GbE/56GbE cards and 100GbE switch - and ofc. 95% of the time using just WiFi

 

[Scott Laird]

FWIW, I don't think there's an easy way to connect a QSFP56 to a QSFP-DD at 200 Gbps today. One is 4 lanes of 50 Gbps, the other is 8 lanes of 25 Gbps. That pretty much rules DAC cables out, so you'd need optics for each that implement the same standard. fs.com only has 3 QSFP56 optics (200gbase-SR4, -LR4, and -FR4) and 2 QSFP-DD optics (both 200gbase-2SR4, which is 2x 100gbase-SR4, not 200gbase-SR4).

It looks like 200G LR4 QSFP-DD modules probably exist, but they're >$1k each and rare.