CX3 with ESXi 7 & Windows Server 2019

[hmw]

The ConnectX-3 is EOS (end of support) since the beginning of 2020. Has anyone gotten it working with ESXi or Windows Server 2019 in Ethernet mode?

I read on the VMware forums that people got it working on ESXi 6.7 with drivers from 6.5. I’m wondering if the same is possible with ESXi 7.0 - being able to use the older nmlx4 driver for 6.5 ...

 

[i386]

Connect-x3 is recognized out of the box with windows server 2019. Install the mellanox drivers/ofed package for more options/tuning tools etc.

 

[klui]

ESXi 7 recognizes it from a fresh install.

 

[cyrus104]

I have several of these cards in ESXi 7.0 machines and having problems getting data over them. They are detected as dual port cards and the link comes up as 56gbps using a SX6036 and FDR QSFP cables.

I think I have these cards full updated to the latest firmware.

I live booted these machine to a linux ubuntu and these cards work as expected. I know it's not much to go on, looking for some thoughts on troubleshooting.

 

[hmw]

I have several of these cards in ESXi 7.0 machines and having problems getting data over them. They are detected as dual port cards and the link comes up as 56gbps using a SX6036 and FDR QSFP cables.

I think I have these cards full updated to the latest firmware.

I live booted these machine to a linux ubuntu and these cards work as expected. I know it's not much to go on, looking for some thoughts on troubleshooting.

I started using Connect4-LX cards, they're supported by 7.0 with SR-IOV.

 

[cyrus104]

I have a SX6036 that supports 56Gbps so going with the LX cards wouldn't be fully compatible based on the mellanox comparison chart.

 

[kapone]

So..uhh...I gotta ask. WTF is the problem here??

They are detected as dual port cards and the link comes up as 56gbps using a SX6036 and FDR QSFP cables.

That's exactly what they are supposed to do, aren't they?

I started using Connect4-LX cards, they're supported by 7.0 with SR-IOV.

And what does that have to do with CX3 cards??

- Are CX3 cards supported in ESXI 7.0? Depends on your definition of "supported". Do they work? Yup.
- Do CX3 cards support SR-IOV? = Yes.
- DO CX3 cards support SR-IOV in ESXI 7.0? Yup.

I repeat, what's the question here?

 

[hmw]

- DO CX3 cards support SR-IOV in ESXI 7.0? Yup.

Really? CX3 (and CX3 Pro) supports SR-IOV in ESXi 7?

 

[kapone]

Really? CX3 (and CX3 Pro) supports SR-IOV in ESXi 7?

I never made the distinction between CX3 and CX3 Pro. If your CX3* card supports SR-IOV, it'll support SR-IOV in ESXI 7.0

 

[cyrus104]

So..uhh...I gotta ask. WTF is the problem here??

The first sentence comes before the second. Even though the cards are detected, there is no data being passed over them. If I put them in a vswitch and remove one of my intel 10G vmnics as the standby no data gets passed. This was easiest for me to test when I the CX-3 on the vmk0 (management nic) vswitch and then removed the other nic. The host appeared to go offline for about 5 minutes before there was an automatic revert which put the other nic back into the vswitch and the host came back up.

I never made the distinction between CX3 and CX3 Pro. If your CX3* card supports SR-IOV, it'll support SR-IOV in ESXI 7.0

The CX3 DO NOT support SR-IOV in ESXI 7.0, this was supported in 6.7 and can also be used with the OFED driver that doesn't work in 7.0. As part of my troubleshooting with the no data being processed, I came a crossed vmware page that specifically stated the issue. I'll go through my browser history and look for the specifically.