ESXi 6.5u3, 2xCX-2 within 2node IBM X3950x5 verses DELLs

[Derwood]

Hey folks,

I was wondering if anyone has had a similar issue, yes a little dated on two cards (ain't rich).

Heres the scenario, three exact same ESXi6.5u3 installs within either a DELL R810, CS24-NV7 using the Mellanox MT26428 (dual QSFP) IB cards all with the latest firmware are working absolutely fine yet the same pair of cards inserted within the IBM X3950x5 2node is having none of the exact same install yet are visible but not active within the manage > Hardware > PCI devices.

Anyone facing similar issues, watchout for M5015 RAID controller under ESXi 7b as not recognised during initial install too as have just tried.

Considerations or maybe possible fixes at a reach is removing a single Mellanox MT26428 from within the 2node IBM, manual labor though and at 6:09zulu is not looking likely!

 

[Derwood]

Another notion maybe I should let you know about is the ESXiCLI, all the DELL machines at terminal level responded well to these commands but I had to change some slight syntax to get the same output within the IBM X3950x5 2node..

DELLs:
esxicli system module set --enabled=false --module=nrdma

Yet the IBMs will accept nothing else then a slight syntax change which is below.
esxicli system module set -m nrdma -e false

I reckon as either the IBM has two boards but no conflicts listed plus the added variable of accepting the same syntax somewhere along the line is preventing me from utilising the cards.

I'll copy and paste the terminal outputs if needed, not had any warnings nor errors.

 

[Derwood]

Here is the IBM X3950x5 2node terminal output, I made sure whatever previous would not hinder the newer install, everything went just as well here as it did within the DELLs, alas, still no joy!

*I also rebooted at every possible step to make sure while within the IBM X3950x5 2node both MT26428 cards are noticed at locations 0000.13.00.0 & 0000.90.00.0, they are spread also between the two interlinked nodes and both reside within the same location relative to one another.

login as: root
Using keyboard-interactive authentication.
Password:
The time and date of this login have been sent to the system logs.

WARNING:
   All commands run on the ESXi shell are logged and may be included in
   support bundles. Do not provide passwords directly on the command line.
   Most tools can prompt for secrets or accept them from standard input.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@localhost:~] esxcli system module set -m nrdma -e false
[root@localhost:~] esxcli system module set -m nrdma_vmkapi_shim -e false
[root@localhost:~] esxcli system module set -m nmlx4_rdma -e false
unknown module 'nmlx4_rdma'
[root@localhost:~] esxcli system module set -m vmkapi_v2_3_0_0_rdma_shim
-e false
[root@localhost:~] esxcli system module set -m vrdma -e false
[root@localhost:~] esxcli software vib remove -n net-mlx4-en
[NoMatchError]
No VIB matching VIB search specification 'net-mlx4-en'.
Please refer to the log file for more details.
[root@localhost:~] esxcli software vib remove -n net-mlx4-core
[NoMatchError]
No VIB matching VIB search specification 'net-mlx4-core'.
Please refer to the log file for more details.
[root@localhost:~] esxcli software vib remove -n nmlx4-rdma
[NoMatchError]
No VIB matching VIB search specification 'nmlx4-rdma'.
Please refer to the log file for more details.
[root@localhost:~] esxcli software vib remove -n nmlx4-en
[NoMatchError]
No VIB matching VIB search specification 'nmlx4-en'.
Please refer to the log file for more details.
[root@localhost:~] esxcli software vib remove -n nmlx4-core
[NoMatchError]
No VIB matching VIB search specification 'nmlx4-core'.
Please refer to the log file for more details.
[root@localhost:~] esxcli software vib remove -n nmlx5-core
[NoMatchError]
No VIB matching VIB search specification 'nmlx5-core'.
Please refer to the log file for more details.
[root@localhost:~] cd /var/log/vmware
[root@localhost:/var/log/vmware] esxcli network firewall set --enabled false
[root@localhost:/var/log/vmware] wget https://XXXX.com/MLNX.zip
Connecting to XXXX.com (XXX.XXX.XX.XXX:443)
MLNX.zip             100% |********************************|  552k
0:00:00 ETA
[root@localhost:/var/log/vmware] esxcli software vib install -d
/var/log/vmware/MLNX.zip
Installation Result
   Message: The update completed successfully, but the system needs to
be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: MEL_bootbank_net-ib-cm_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_net-ib-core_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_net-ib-ipoib_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_net-ib-mad_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_net-ib-sa_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_net-ib-umad_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_net-memtrack_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_net-mlx4-core_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_net-mlx4-ib_1.8.2.5-1OEM.600.0.0.2494585,
MEL_bootbank_scsi-ib-srp_1.8.2.5-1OEM.600.0.0.2494585
   VIBs Removed:
   VIBs Skipped:
[root@localhost:/var/log/vmware]

Below (I reckon) should or are two screenshots of the IBM 2node web control interfaces.

 

[Derwood]

Attached here, is the DELL R810 pNIC view which I might add are all total clean fresh installs.


Im really considering moving all CX-2 cards to Chelsio T580 which allows for 40GbE ethernet rather then using the IB for IPoIB usage, but still utilising the same switch.