Nvidia Connectx-9

[jpmomo]

Does anyone have any hands-on experience with these nics? One of the differences between the cx8 and cx9 is supposed to be a single 800GE logical port when switched into ethernet mode. the cx8 can only be configured with 2x400GE.
When running either of these nics in a pci gen5 server, you need to use the mcio aux connector to get an additional 16 pci gen5 lanes. this is needed to run at the full 800Gbps. When you do this, it will create 2 different pci id for each pci "slot".
The cx8 when put in IB mode, will use 200G (224G) serdes vs the 100G in Ethernet mode. The cx9 is supposed to support the 200G lanes in ethernet mode as well.
I am looking for some confirmation on both of these assumptions.

 

[i386]

Cx8 and cx9 are way Out of my Budget, at Home and at Work

 

[Stephan]

Cx8 and cx9 are way Out of my Budget

800GE is more than one full 50 GB bluray, every second. Doubt anyone on here has touched one of these, maybe @Patrick at some expo, for five seconds, for a PR picture.

 

[blunden]

800GE is more than one full 50 GB bluray, every second. Doubt anyone on here has touched one of these, maybe @Patrick at some expo, for five seconds, for a PR picture.

Yeah, either STH staff or perhaps someone has access to them at work. I doubt anyone is using them in their home lab.

 

[miraculix]

Does anyone have any hands-on experience with these nics?

I don't have hands on experience with CX9. My CX8 experience is with B300 (C8180P) and very recently GB300 (mezzanine not sure what CX8 model) both PCIe 6. I've only done research for CX8 on PCIe 5 systems which hasn't been needed yet, but it's pretty straightforward.

One of the differences between the cx8 and cx9 is supposed to be a single 800GE logical port when switched into ethernet mode. the cx8 can only be configured with 2x400GE.

Correct. For example the C8180P I'm most used to is OSFP. It can do XDR 800Gb/s for IB but has to port split to 2x400GbE or 8x100GbE. Similar with the other CX8 cards. Use case is typically 2x400GbE into a dual-plane Ethernet backend fabric. Lines up nicely with dual MPO-12 800GBASE-DR8 on both node and switch sides.

CX9 can do 800GbE without port split.

When running either of these nics in a pci gen5 server, you need to use the mcio aux connector to get an additional 16 pci gen5 lanes. this is needed to run at the full 800Gbps. When you do this, it will create 2 different pci id for each pci "slot".
The cx8 when put in IB mode, will use 200G (224G) serdes vs the 100G in Ethernet mode. The cx9 is supposed to support the 200G lanes in ethernet mode as well.

Take a look at the PCIe Auxiliary Card Kit details for CX8 and CX9. MCIO cable joins the CX8/CX9 card to the PCIe x16 aux card kit. Correct on two different PCI IDs. Be sure to check details like which SKUs support Socket-Direct vs DSP, DSP to BF for CX9 etc.