vSphere 7.0 and 10Gbe

[muhfugen]

Perhaps it is necessary to start using PCIe passthrough of the old devices to feature VM's with older OS's, like a storage VM with freenas or a network VM for external networking.

Sadly even if you tag a device for pci passthru, it still constantly bitches and moans about how it is unsupported. i have a few Xeon-d boards which have onboard 16 port lsi 2008s which i use for storage spaces direct/scale out file server, and pass them through to the VMs, and still have to click to install with unsupported hardware everytime i install a vSphere patch. its annoying as hell.

 

[Rand__]

I have several of these cards in ESXi 7.0 machines and having problems getting data over them. They are detected as dual port cards and the link comes up as 56gbps using a SX6036 and FDR QSFP cables.

You might want to elaborate on the kind of problems you have/ what issues you see?

They still are on the compatibility list, so I would not expect to see major issues...




O/c it depends on how you use them - Host Networking (has to work) vs VF's (I think it got removed) vs pass through (might)...

 

[cyrus104]

Thanks for highlighting this.

I have 3 ESXi hosts each with a CX-3 dual port card. These are connected to an SX6036 switch that is in ethernet only mode right now, I didn't have much need for IB until I move on to play/testing that. All of the connections are made with Mellanox QSFP to QSFP FDR rated cables.

In each host they both come up as expected, vmnic8 and vmnic1000802 (because mellanox uses one address, vmware gives it a name based on pcie addressing). Auto-negotiation doesn't really work on the SX6036 so I have to manually set speeds. For testing I set one to the accepted normal rate of 40Gbps (vmnic8) and I set the other one to a Mellanox proprietary 56Gbps which is supported with by the switch, nic, and cable.



Whenever I add either CX-3 nic to a vSwitch specifically 0 where my management vmk0 is, I get an error. The host disconnects and after about 2 minutes the host does a roll back and everything is back to normal.



I am running at 9k MTU and I've made sure that everything has that set.

For vSwitch0, I have vmnic6 a 10Gbps sfp going into the same SX6036 as the vmnic8. When I remove vmnic6 (10Gbps) is when I get the error and roll back. So I know the switch is performing vlan tagging and data passing at 10Gbps just fine.



Thanks for any help.

 

[Rand__]

1. I usually leave ESXi on AutoNeg and configure desired speed via the switch/interface, that works fine for the CX3.
2. Your issue occurs when you change the current management vSwitch? Thats not indicative that the NIC has issues, its quite normal for ESXi to be finicky when you modify that (active) vswitch.

3.To test the NIC: Create a new switch with a vmk, add an IP in a new range (new vlan if you want to), add the nic to that. Repeat on a second host. Adjust vlan on switch if necessary. Test.
If that works, its not ESXi, but your messing with the management vswitch



Btw why are you not using a dvSwitch if you run 3 hosts? I assume youre going for a cluster setup; dvSwitch will make things easier for that (most of the time)


I usually do sth like the following when setting up a new box and trying to change management from the onboard nic (and dhcp ip to the MLXs and permanent ip). Basically add a new switch, make it management capable, add the new ip, add new nic, add a backup nic to the old one on another vlan [in case its not working as expected], remove old nic, reboot and hope the new nick takes.
Maybe it helps.

myHOSTNAME=esx
myIP=a.b.c.d

esxcli system hostname set --host=$myHOSTNAME
esxcli system hostname set --fqdn=$myHOSTNAME.myDomain

esxcli network vswitch standard add --vswitch-name=vS_MGMT
esxcli network vswitch standard portgroup add --portgroup-name=pG_MGMT  --vswitch-name=vS_MGMT
esxcli network vswitch standard portgroup add --portgroup-name=1G_Failover  --vswitch-name=vS_MGMT

esxcli network ip interface add --interface-name=vmk20 --portgroup-name=pG_MGMT   

esxcli network ip interface ipv4 set  --interface-name=vmk20 --ipv4=$myIP --netmask=255.255.255.0 --type=static

esxcli network ip interface tag add -i vmk20 -t Management

esxcfg-route -a default GWIP;esxcli network ip interface set -e false -i vmk20; esxcli network ip interface set -e true -i vmk20



esxcfg-nics -l #show physical nics + mtu
esxcli network ip interface list # show vmks + mtu
esxcli network ip interface ipv4 get #show vmk IPs

#--->Remove vmnic0 from Default network first and ensure accessibility (ie add vmnic 2 as uplink to vm network + CABLEING/VLAN check (if vmnic1!!))
esxcli network vswitch standard uplink add --uplink-name=vmnic2 --vswitch-name=vSwitch0
esxcli network vswitch standard uplink remove --uplink-name=vmnic0 --vswitch-name=vSwitch0
esxcli network vswitch standard uplink add --uplink-name=vmnic0 --vswitch-name=vS_MGMT  ##########Nur als STBY drin?

#accessibile via new IP

esxcli network vswitch standard add --vswitch-name=vS_DMZ
esxcli network vswitch standard portgroup add --portgroup-name=DMZNet  --vswitch-name=vS_DMZ
esxcli network vswitch standard uplink add --uplink-name=vmnic1 --vswitch-name=vS_DMZ

#--->2 DO:  ensure accessibility via new IP 
#---> Then remove vmnic1 Default network if added and add to vs_DMZ # else vmnic 1 removal will casue error - ignore

#Delete VIA GUI at this time
esxcli network vswitch standard uplink remove --uplink-name=vmnic2 --vswitch-name=vSwitch0
esxcli network vswitch standard uplink remove --uplink-name=vmnic128 --vswitch-name=vSwitch0
#-----NOW RECONNECT TO NEW IP

# reboot needed on freeze, new ip should continue to work

esxcli network ip interface  remove  --interface-name vmk0
esxcli network vswitch standard  remove --vswitch-name=vSwitch0

 

[cyrus104]

1) Check - I'm leaving it on AutoNeg now. The SX6036 doesn't really do AutoNeg so I set it to a speed and then have see that vmware does pickup either 40Gbps or 56Gbps.
2) Yes, when I change vSwitch0 I have a management Network and a VM Network. I do have a dvSwitch right now for vMotion.
3) I "think", I successfully tested this by creating a new dvSwitch and put the specific port in question as the uplink and was able have a vm connect to the internet successfully. With a vmxnet vm interface, the vm shows a 10Gbps interface. It's also pretty slow when actually running iperf3.

I don't fully use dvSwitches because.... I don't fully understand them. I have a default dvSwitch for vMotion that has 1x 10Gbps uplink right now, it has it's own vlan.

I'm reading though your example and I think I can have a management vswitch or add it to a dvswitch. I will then just use 1x of the CX-3 interfaces for normal vm connectivity and then the other interface from the dual card for the vmotion vlan. With have 2x 56gbps and 2x 10gbps, I just need to figure out the best combination of vswitches vs dvswitch and the layout of those. So confusing, I'm use to XCP-NG and Proxmox.

 

[Rand__]

iperf3 is single threaded an not really a good maximum perf test. There is a myriad of threads on this forum with alternatives and options

My example is for regular vswitches only (as dvswitches can't be managed from esxi cli unfortunately);
I run a dvswitch primarily and vswitches only for backup (management) and segregation reasons (dedicated dmz interface).

My dvSwitch Layout for my vsan Cluster (based on some ancient best practices for high speed (10G) esxi clusters)

 

[cyrus104]

Thanks for the advice on iperf3, I'll lookup some of those threads to make sure I'm actually gettting the performance I'm expecting to get... or close.

Ok, I've got a working dvswitch using the interface that was seeming to cause issues. I can see that the craziness will be how to have different one DSwitch with many DPortGroups like you have or having mutliple DSwitches with fewer port groups. I guess the number of 56Gbps interfaces will restrict that for me.

Anyways, so far so good.