vSphere 7.0 and 10Gbe

[RTM]

Back on topic:
So if I am reading the information in the HCL correctly, OP's card should still be supported, just by a different driver now (nmlx vs the old mlx):
VMware Compatibility Guide - I/O Device Search

 

[marcoi]

any updates with MCX311A-XCAT working on esxi 7 yet?

 

[namike]

any updates with MCX311A-XCAT working on esxi 7 yet?

I'm going to be testing later today, waiting on USPS to show up with my card.

Edit: Actually tomorrow now as USPS tagged on another day.

 

[marcoi]

@namike any update on the card working on esxi7?

 

[namike]

USPS finally showed up this afternoon (Monday). Pop'd the card in my Win10 workstation and updated the FW. Shutdown my ESXi box and loaded ESXi 7 to a new thumbdrive. The ConnectX-3 is being detected right out of the box.

 

[marcoi]

Thank you, ordering a few now

 

[Dev_Mgr]

One option is the Broadcom 57810 NICs. They are available in SFP+ and RJ45 ported models. They are common options in Dell serves and still supported with ESXi 7.0 (they are now listed in the HCL under Qlogic as Qlogic bought the network division of Broadcom). Both the SFP+ model and the RJ45 model can be found on Ebay for less than US$40.

Note: I don't have any HP/HPE servers in my home lab to test if these cards will work in one of those servers.

 

[StevenDTX]

I ended up replacing all of my 10Gb NICs with HP 560SFP+, which are Intel X520-DA2s.

 

[vangoose]

One option is the Broadcom 57810 NICs. They are available in SFP+ and RJ45 ported models. They are common options in Dell serves and still supported with ESXi 7.0 (they are now listed in the HCL under Qlogic as Qlogic bought the network division of Broadcom). Both the SFP+ model and the RJ45 model can be found on Ebay for less than US$40.

Note: I don't have any HP/HPE servers in my home lab to test if these cards will work in one of those servers.

I have both 57810 amd CX-3 in my 6.7U3, may try to load 7.0 and see but don't have plan to do full upgrade yet.

The good thing about 57810 is you can carve out up to 4 physical functions out of a single nic. A dual port 57810 will give you 8 nics in esxi. I use 57810 for front end and guest and cx-3 for storage.

My 57810 is hpe version, make sure you update firmware if you want to use it.

 

[zer0sum]

I can confirm that my Mellanox ConnectX-3 cards still seem to work just fine

They are HP 649281-B21 that have been reflashed to Mellanox MCX354A-FCBT firmware

@marcoi - These run for $25-35 on Ebay and will do 10/40/56G with 2 x QSFP+ ports

 

[Rand__]

Qsfp+

 

[zer0sum]

Here are the lists from Vmware:

• 7.0 native and removed in driver deprecated devices
• 7.0 VMKlinux deprecated devices
• Deprecated Devices 7.0 native and not removed in driver

 

[WANg]

Mellanox native ConnectX-3 drivers (nmlx4) seemed to have its SRIOV functionality removed compared to its VMKLinux/OFED equivalent.
If you need virtual functions for your VM, you might want to verify its functionality or hold off on a 7.0 upgrade/install.

Chelsio T520/540/580 (Terminator 5 series) 1/10/40GbE cards are also supported by its ESXi native (7.0 kosher) driver and should be less than 100 each, especially in its Dell/HP branded guises. The same might go for the Solarflare XtremeScale (SFN8522) cards and such.

Chelsio Terminator 4 series (T420 and below) and the Solarflare Flareon 5/6/7 series (SFN 5xxx/6xxx/7xxx) cards are all deprecated with no VMKLinux support - that's not mentioned in the lists.

 

[Net-Runner]

In the case when RDMA/RoCE is not mandatory and only bandwidth matters, you could potentially pass through the unsupported network adapter to an OpenWRT or pfSense virtual machine and route it back down to ESXi over regular VMxnet. That is how a WiFi adapter for a small ESXi host works in my home lab currently.

 

[zer0sum]

Mellanox native ConnectX-3 drivers (nmlx4) seemed to have its SRIOV functionality removed compared to its VMKLinux/OFED equivalent.
If you need virtual functions for your VM, you might want to verify its functionality or hold off on a 7.0 upgrade/install.

Chelsio T520/540/580 (Terminator 5 series) 1/10/40GbE cards are also supported by its ESXi native (7.0 kosher) driver and should be less than 100 each, especially in its Dell/HP branded guises. The same might go for the Solarflare XtremeScale (SFN8522) cards and such.

Chelsio Terminator 4 series (T420 and below) and the Solarflare Flareon 5/6/7 series (SFN 5xxx/6xxx/7xxx) cards are all deprecated with no VMKLinux support - that's not mentioned in the lists.

It does look to me like CX3 cards are unsupported for SR-IOV at this time.
Now I've gotta find a new sub $30 NIC for my home lab boxes

VMware Compatibility Guide - I/O Device Search

www.vmware.com

 

[vangoose]

It does look to me like CX3 cards are unsupported for SR-IOV at this time.
Now I've gotta find a new sub $30 NIC for my home lab boxes

VMware Compatibility Guide - I/O Device Search

www.vmware.com

It's removed from 6.7.
BCM57810 is very cheap and still has SR-IOV supported on 7.0. Check Guest OS support.

 

[zer0sum]

It's removed from 6.7.
BCM57810 is very cheap and still has SR-IOV supported on 7.0. Check Guest OS support.

I'll take a look at those. Thank you

And it seems anything with the good old Intel 82599 controllers is still supported for SR-IOV in esxi 7.0 from what I can tell
I know you can find the SuperMicro AOC-STGN-I2S for ~$35

 

[vangoose]

I'll take a look at those. Thank you

And it seems anything with the good old Intel 82599 controllers is still supported for SR-IOV in esxi 7.0 from what I can tell
I know you can find the SuperMicro AOC-STGN-I2S for ~$35

Intel works very well and has best guest os support for vf driver.

BCM has npar that intel doesn't have. You can carve the physical nic into 4 physical functions, then up to 16 virtual function per PF per port.
With dual port, you can have 8 physical nic in ESXi. I use
- 2 for ESXi management in VSS
- 2 for vMotion/NFS, etc. in VDS
- 2 for iSCSI in VDS
- 2 for guest in VDS

 

[WANg]

It's removed from 6.7.
BCM57810 is very cheap and still has SR-IOV supported on 7.0. Check Guest OS support.

It's not removed from 6.7 - it's removed from 6.0. It was a subject of an old thread here. nmlx4 and SRIOV just isn't a "thing" - to make SRIOV work in 6.0-6.7, you'll need to disable/blacklist the nmlx4 driver and fall back on OFED (VMKLinux).

Note that SRIOV functionality still works (and works great) on Linux and FreeBSD, so don't toss that ConnectX3 card just yet - it's still perfectly useable in other platforms (like Proxmox)...just not in VSphere 7 and above.

 

[cyrus104]

I have several of these cards in ESXi 7.0 machines and having problems getting data over them. They are detected as dual port cards and the link comes up as 56gbps using a SX6036 and FDR QSFP cables.

I think I have these cards full updated to the latest firmware.

I live booted these machine to a linux ubuntu and these cards work as expected. I know it's not much to go on, looking for some thoughts on troubleshooting.