Upgrade to ESXi 5.5 with C6100 and Mellanox

[markpower28]

I have ESXi 5.1 on C6100 running very stable with Mellanox ConnectX 2 VPI for some time now (driver OFED 1.8.1) and decide it's time to upgrade to vSphere 5.5a.

First, I tried update manager. It shows the driver on 5.1 (1.8.1) is not compatible with 5.5 (it comes with 1.9.7). Then I tried use Patrick's method using ESXi-Customizer replace 1.9.7 drive with 1.8.2 driver. It has couple warnings but I am able to create the ISO. During the install, it stopped at driver components not find. Looks like upgrade is not a good option at this point.

Then, I decide to using 5.5 ISO for a fresh install. ConnectX 2 show up fine but it only has Ethernet driver not IB driver. Update manager does not work in here either since it already has the 1.9.7 driver which is newer then 1.8.2. The following article saved my day InfiniBand install & config for vSphere 5.5 | Erik Bussink

I did the following for the fresh loaded 5.5 on C6100

1. enable shell and SSH
2. Remove 1.9.7 driver by the following command: esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core
3. reboot
4. verify 1.9.7 is gone for good: esxcli software vib list | grep mlx4
5. use WinSCP upload 1.8.2 zip file to tmp folder
6. install 1.8.2 by the following command: esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.2.0.zip --no-sig-check
7. Reboot

Then everything works just like before

The only problem for production environment is, in-place upgrade does not seems like an option until newer version of OFED for ESXi is available.

 

[Fzdog2]

I have seen some weird issues with 5.5 on my C6100. 2 of my nodes are frequently unable to get past the IMPI part of the boot up process. I have to reflash the BMC to the latest version and then leave them running. After awhile, if I try a reboot or cold boot, it stops loading the IMPI module again.

 

[markpower28]

have you try the following/ https://communities.vmware.com/thread/460763

 

[Fzdog2]

I've seen that process, but have not tried it. Wouldn't uninstalling the IPMI VIB disable any remote power control through ESXi?

 

[NetWise]

I had similar IPMI problems all through the 5.5 beta period. In the end it turned out to be some old sketchy SATA disks that were 'thinking' about failing. Support had me pop them out and problems went away. If you can try booting from USB or disk less and see if that helps at all.

 

[Fzdog2]

I do boot from USB, but I have 1 SSD and 1 HDD in each node for VSAN use. The SSD's are brand new, but the HDD's are 3+ year old WD Green drives. Maybe I should run some drive checks.