Nvidia CX-7 400G NIC --- QSFP-DD to OSFP DAC?

[jpmomo]

STH just posted an article on these nic cards.

I am trying to get some clarification on the specific cable to connect a QSFP-DD switch to one of the OSFP cx-7 nics.
There are 2 issues that we encountered:

1. the nic side requires an OSFP "Flat top" connector not the "Fin top" that is normally found on OSFP cables.
2. QSFP-DD uses 50G lanes vs OSFP's 100G lanes.

We were told by the folks at Nvidia that we would need a combination of 2 different transceivers and a fiber cable to connect the switch to the nic.

In the STH article, they mention this issue but I could not discern the exact cable they wound up using.
They also mention that they had a DAC that had QSFP-DD on one side and OSFP on the other. The OSFP side was probably Fin topped and would not physically fit into the nic. I am also curious how the DAC would work even if it was Flat topped on the OSFP side. How does the DAC convert 50G electrical lanes to 100G lanes that the NICs supposedly require?

DACs are relatively cheap compared to transceivers with a fiber cable.

There was a new member that posted on this forum a while ago that had one of these NICs. He ran into this issue when trying to find a cable that would work.
I think he physically remove the heatsink from the cage on the nic! That might have allowed the standard OSFP finned top dac to fit.
I did not know what he was trying to connect the other end of the cable to (switch or nic).
He didn't post or reply after that first thread so not sure of the outcome.


The DAC cables that I am able to use successfully are OSFP800 Flat top on both sides. In that use case I am connecting NIC to NIC. Both sides are OSFP and both NICs use 100G lanes.

 

[FIRAT]

Hi,

Can you share the details of the optics you were provided by Nvidia to do this?

We were told by the folks at Nvidia that we would need a combination of 2 different transceivers and a fiber cable to connect the switch to the nic.

We are looking to bridge HDR IB Switches (QSFP56) with NDR IB Switches (OSFP112), and ordered the DAC cables for this through Nvidia already, but they have substantial lead-times. QSFP-DD seeming has won that battle on the Ethernet side, and OSFP is fading away, therefore finding 3rd party cables to do this is also near impossible (FS, Siemon, misc. other 3rd parties we deal with do not carry these).

In the interim, going off what Patrick referred to as "funky optics" on the

, I'm assuming a combination of QSFP56 or QSFP28 MPO optics on one end and OSFP MPO on the other end may work? The same issue you raised, however, applies of encoding & speed differences, which is why I'm asking for the details on the optics, and any other related info you can share.

My understanding is that:

• NDR w/ OSFP does 8 lanes of 50G/lane with PAM4 encoding for 400G
• HDR w/ QSFP56 does 4 lanes of 50G PAM4 for 200G
• EDR w/ QSFP28 does either 4 lanes of 25G NRZ or 2 lanes of 50G PAM4 for 100G

So for optics, as long as wave-length/modulation matches e.g., SR4 at 850nm , as well as the connector, the compatibility should boil down to matching encoding and number of lanes? Going between generations (400G -> 200G) matching the lanes 1-for-1 won't work though, so do you have any idea how that is handled? e.g., 4 of the 8 lanes are simply not used, and the OSFP switchport down-negotiates at 200G instead of 400G?

 

[jpmomo]

I am not exactly following what you need to accomplish.

Originally, I was trying to connect a qsfp-dd (400G) device to a single port osfp cx-7 nic.

that required several expensive and hard to get components:

mms1v00-wm (400G DR4 single mode transceiver) mfp7e30-nxxx (single mode crossover fiber mpo-12/apc to mpo-12/apc) then mms4x00-ns400 (osfp flattop 400G single mode transceiver)

real straight forward......not!!!

My use case did not require a single 400G port on the nic side. So we could use the dual port 200G version of the cx-7 nics.
Those nics used qsfp112 interfaces (and worked properly with qsfp56 dacs when connecting b2b)

Theoretically, you should be able to use a qsfp-dd fanout to 2 x qsfp56 dac to connect the qsfp-dd device to the dual port cx-7 nic.

The qsfp-dd device needs to be able to split its 8 50G lanes into 2x4x50G. The dual port cx-7s appear to correctly connect via the 4x50G lanes for 200G each port.

You can get one of those dacs cheap and quick.

Nvidia does have 1 sku that is a single port 400G qsfp112. that would be the mcx715105as-weat if you can find one. it also appears to be listed as engineering sample and doesn't have its firmware listed on their public website. You still have the issue of converting 50G lanes on the qsfp-dd/switch side to 100G lanes on the nic side.

The osfp route on the nic side is the tricky one. especially when you consider the "flat top" issue on the transceiver.

PM me if you need to discuss in more detail.

 

[FIRAT]

Hi Thank you for the detailed reply!

I was trying to connect their NDR 32-port 800gbps per OSFP port switch to a HDR 40-port 200gbps per QSFP56 switch ~1.2km away. To this end, I was using their 200gbps 2km optics (mms1w50-hm) which are QSFP112, and so I was looking for an adapter to convert OSFP to QSFP. I was able to find and buy a OSFP to QSFP28 adapter, which works via any old 100 gbps QSFP28 DAC to connect the NDR switch to a HDR switch at up to 5 meters away.

When they arrive, the NDR-to-HDR cables from Nvidia, I'm sure will also work to step-down the 800gbps to 200gbps, effectively doubling the available bandwidth.

However, as both of these solutions require an additional HDR switch, sitting next to the NDR equipment, they're sub-optimal. As currently the topology is NDR -> HDR -> 2km optics -> HDR, whereas the desired topology is NDR -> OSFP-QSFP adapters -> 2km optics -> HDR.

I haven't had a chance to test whether the OSFP-to-QSFP28 adapter works on the NDR switch with a 100gbps 2km optic (MMA1L10-CR), which will be next, followed shortly with whether they'll work with a 2km 200gbps optic (somewhat doubtful given the QFSP28 specification).

Also, one correction on my original statement:

• NDR w/ OSFP does 8 lanes of 50G/lane with PAM4 encoding for 400G

This is not correct, NDR w/ OSFP does 8 lanes of 100G/lane with PAM4 encoding for 800G, and the port is split into 2, each doing 4x 100 G lane.