PSOD Woes

Zack Hehmann

Member
Feb 6, 2016
67
5
8
All I have been receiving PSOD errors on one of my hosts at home. This has been going on for about a month. Usually the host will stay up for 1 day and 15 hours before doing a kernel dump. I reduced the number of VMs running to just the vCenter server and it still crashes.

The error is PF Exception 14 in world 6xxxxxx and lists Intel-nvme-c or CmdComp1 in the error.

Today I ended up migrating all of the VMs off of that datastore to an identical NVMe drive that I just installed. Both are Intel DC3700 1.6 TB AICs. The host is a Dell r620 with dual e5-2670 and 384GB of RAM.

I have had the host for about 2-3 months. It's running ESXi 6.5 and I installed vCenter server 6.7 about 1.5 months ago. I also changed the CPUs from 2620 to the 2670 about a month ago.

I read a few things about the error and read through the log files. I think it's related to the NVMe drive.

I can provide the logs and screenshots later when I'm not on my cellphone.

Recommendations on where to host the screenshots and dump files so I can link/attach those?

Thanks for your help.
 

Rand__

Well-Known Member
Mar 6, 2014
5,162
1,070
113
Code:
 esxcli software vib list |grep nvme
nvme                           1.2.2.27-1vmw.670.2.48.13006603       VMW     VMwareCertified   2019-05-11
vmware-esx-esxcli-nvme-plugin  1.2.0.36-2.48.13006603                VMware  VMwareCertified   2019-05-11
Driver, eg from intel - note the one below is older
Code:
esxcli software vib install -v /vmfs/volumes/freenas/nvme/VMW-ESX-6.7.0-intel-nvme-vmd-1.4.0.1016-8733247/intel-nvme-vmd-1.4.0.1016-1OEM.670.0.0.8169922.x86_64.vib
 

marcoi

Well-Known Member
Apr 6, 2013
1,411
226
63
Gotha Florida
Can you get a 2TB SSD drive and run off that for a week to see if its your nvme device?
Also have you tried running the server with no vm running to see if it crashes over 2 days? Have you checked the temps on the components to make sure the aic isnt overheating? Maybe set a fan on it for a few days to see if extra air solves the issue?

Did you try moving the aic to another slot? maybe your current slot has issues or isnt a high enough speed for the aic, etc.
 

StevenDTX

Active Member
Aug 17, 2016
440
148
43
Also, can you pull an IPMI/DRAC logs?

I had an issue with PSOD a while back, and IPMI logs showed a possible bad DIMM.
 

Zack Hehmann

Member
Feb 6, 2016
67
5
8
Code:
 esxcli software vib list |grep nvme
nvme                           1.2.2.27-1vmw.670.2.48.13006603       VMW     VMwareCertified   2019-05-11
vmware-esx-esxcli-nvme-plugin  1.2.0.36-2.48.13006603                VMware  VMwareCertified   2019-05-11
Driver, eg from intel - note the one below is older
Code:
esxcli software vib install -v /vmfs/volumes/freenas/nvme/VMW-ESX-6.7.0-intel-nvme-vmd-1.4.0.1016-8733247/intel-nvme-vmd-1.4.0.1016-1OEM.670.0.0.8169922.x86_64.vib

Here is the output from my host.

Code:
[root@localhost:~]  esxcli software vib list |grep nvme

intel-nvme                     1.3.2.8-1OEM.650.0.0.4598673          INT                    VMwareCertified   2019-05-03  
mx-nvme                        1.0.50.0-1OEM.600.0.0.2768847         MAN                    VMwareCertified   2019-05-03  
smgnvme                        1.0.0.0-1OEM.650.0.0.4240417          SMG                    VMwareCertified   2019-05-03  
nvme                           1.2.1.34-1vmw.650.2.50.8294253        VMW                    VMwareCertified   2019-05-03  
vmware-esx-esxcli-nvme-plugin  1.2.0.32-2.50.8294253                 VMware                 VMwareCertified   2019-05-03
 

Rand__

Well-Known Member
Mar 6, 2014
5,162
1,070
113
Well you seem to have both intel and native nvme drivers running...
I can't find any really recent intel driver though, the latest seems to be 1.5.0.1009 (still significantly newer than yours though)
VMware Maintenance

I'd remove the intel one and go with ESXi native - never have seen a real advantage of the intel one, but had some not so nice issues with them.

Are you on the latest 6.5 release? Any reason not to upgrade/ upgrade to 6.7?
 

Zack Hehmann

Member
Feb 6, 2016
67
5
8
Well you seem to have both intel and native nvme drivers running...
I can't find any really recent intel driver though, the latest seems to be 1.5.0.1009 (still significantly newer than yours though)
VMware Maintenance

I'd remove the intel one and go with ESXi native - never have seen a real advantage of the intel one, but had some not so nice issues with them.

Are you on the latest 6.5 release? Any reason not to upgrade/ upgrade to 6.7?

I was able to find this VMware ESXi 6.5 intel-nvme-vmd 1.8.0.1001 NVMe Driver for Intel dated 2019-06-24

I feel like several things that I changed that were mentioned in my first post. e.g. cpu upgrade, vCenter server install, e.t.c. so I'm not trying to change a bunch of things all at once. I remember trying 6.7 a while back and didn't like it, so that was the reason for picking 6.5 when I newly deployed this host a few months ago.

I'm only trying to change one thing at a time so I can figure out what the problem is.


Can you get a 2TB SSD drive and run off that for a week to see if its your nvme device?
Also have you tried running the server with no vm running to see if it crashes over 2 days? Have you checked the temps on the components to make sure the aic isnt overheating? Maybe set a fan on it for a few days to see if extra air solves the issue?

Did you try moving the aic to another slot? maybe your current slot has issues or isnt a high enough speed for the aic, etc.
To the point I was making earlier. I meant to only change one thing, but turns out I changed 3.

First I'm using a different AIC.
Second it's in a different slot, but using the same riser card.
Third I don't have any VMs running, that includes the vCenter server I had running previously.


I have not way of changing the number of fans or putting a fan directly on the AIC since its a 1U server and all the fans are installed and working.

I looked at the vCenter serve and the iDRAC and I wasn't able to see temps on the AIC. Everything show up as good.

I'm sure there are some ways to check the temps using a CLI utility, but I think it might be limited due to running ESXi.

The AIC is a 4X card and its in 16X physical slot. Not really sure about the slots on the Dell is I don't really care for Dell. The 2 hosts I have came form a working environment with the AIC installed. I more than likely will Migrate off of this hardware and move the components to a SuperMicro x9 Mobo, but I'm not sure what I'm going to do yet.


Lastly, the host is still up and running at 1 day and 5.5 hours. I'll keep you posted.

Thanks for everyone's help!
 

Rand__

Well-Known Member
Mar 6, 2014
5,162
1,070
113
Assuming the drives are P3700's then you should be able to install isdct ( intel ssd datacenter tool), that should be able to read temps off your drive
 

Zack Hehmann

Member
Feb 6, 2016
67
5
8
Assuming the drives are P3700's then you should be able to install isdct ( intel ssd datacenter tool), that should be able to read temps off your drive

I copied the latest vib to the ESXi host to the /tmp directory. Following Intel's install guide I get that I don't have enough space on the host?

Code:
[root@localhost:/vmfs/volumes/5d2c2764-c92001c4-6083-90b11c154990] esxcli software acceptance get
PartnerSupported
[root@localhost:/vmfs/volumes/5d2c2764-c92001c4-6083-90b11c154990] esxcli software acceptance set --level=CommunitySupported
Host acceptance level changed to 'CommunitySupported'.
[root@localhost:/vmfs/volumes/5d2c2764-c92001c4-6083-90b11c154990] esxcli software vib install -v /tmp/
intel_ssd_data_center_tool-3.0.19-400.vib  nfsgssd_krb5cc/                            probe.session                              vmware-root/
[root@localhost:/vmfs/volumes/5d2c2764-c92001c4-6083-90b11c154990] esxcli software vib install -v /tmp/intel_ssd_data_center_tool-3.0.19-400.vib
 [InstallationError]
 The pending transaction requires 261 MB free space, however the maximum supported size is 239 MB.
 Please refer to the log file for more details.

So I Googled for "The pending transaction requires 261 MB free space, however the maximum supported size is 239 MB" and I found this VMware KB. It basically says that there isn't a known fix, but you should just delete "some unused" vibs to make room in the bootbank..? Did you guys have to do the same thing? ESXi is installed on a 4GB USB drive. Here is the link of the partitions from the vSphere Web Client.

Here is the output of df -h


Code:
[root@localhost:/vmfs/volumes/5d2c2764-c92001c4-6083-90b11c154990] df -h
Filesystem   Size   Used Available Use% Mounted on
VMFS-6       7.6T 555.3G      7.1T   7% /vmfs/volumes/VHOSTRST003-LocalDS-01
VMFS-6       1.5T 443.3G      1.0T  30% /vmfs/volumes/R620-01-NVME-02
vfat       249.7M   8.0K    249.7M   0% /vmfs/volumes/4bac53e9-ef270697-44cd-c66d87b3b24e
vfat       285.8M 209.1M     76.8M  73% /vmfs/volumes/5ccbb1f7-04a5be60-9433-90b11c188b4b
vfat       249.7M 220.9M     28.8M  88% /vmfs/volumes/18c763e4-898c5a70-9366-b728a4f82b4f
The 18c763e4-898c5a70-9366-b728a4f82b4f volume is the /bootblock directory. It looks pretty full.

I also decided to see what was in the /bootbank directory.
Code:
[root@localhost:~] cd /bootbank/
[root@localhost:/vmfs/volumes/18c763e4-898c5a70-9366-b728a4f82b4f] ls -al
total 225908
drwxr-xr-x    1 root     root           512 Jul 16 19:55 ..
-rwx------    1 root     root         11225 May  3 03:14 a.b00
-rwx------    1 root     root         96441 May  3 03:18 ata_liba.v00
-rwx------    1 root     root          8596 May  3 03:18 ata_pata.v00
-rwx------    1 root     root          6533 May  3 03:18 ata_pata.v01
-rwx------    1 root     root          6792 May  3 03:18 ata_pata.v02
-rwx------    1 root     root          7432 May  3 03:18 ata_pata.v03
-rwx------    1 root     root          8682 May  3 03:18 ata_pata.v04
-rwx------    1 root     root          7317 May  3 03:18 ata_pata.v05
-rwx------    1 root     root          6421 May  3 03:18 ata_pata.v06
-rwx------    1 root     root          7733 May  3 03:18 ata_pata.v07
-rwx------    1 root     root         90402 May  3 03:14 b.b00
-rwx------    1 root     root         24861 May  3 03:18 block_cc.v00
-rwx------    1 root     root        613656 May  3 03:18 block_hi.v00
-rwx------    1 root     root        175717 May  3 03:17 bnxtnet.v00
-rwx------    1 root     root         87328 May  3 03:17 bnxtroce.v00
-rwx------    1 root     root          3172 May  3 03:46 boot.cfg
-rwx------    1 root     root        575047 May  3 03:18 brcmfcoe.v00
-rwx------    1 root     root        506136 May  3 03:18 btldr.t00
-rwx------    1 root     root         14586 May  3 03:18 char_ran.v00
-rwx------    1 root     root         19735 May  3 03:14 chardevs.b00
-rwx------    1 root     root       2496789 May  3 03:17 cxl.v00
-rwx------    1 root     root         90328 May  3 03:17 dell_sha.v00
-rwx------    1 root     root         33707 May  3 03:18 ehci_ehc.v00
-rwx------    1 root     root        156618 May  3 03:18 elxnet.v00
-rwx------    1 root     root          5988 May  3 03:18 emulex_e.v00
-rwx------    1 root     root         95873 May  3 03:18 esx_dvfi.v00
-rwx------    1 root     root       3550520 May  3 03:18 esx_ui.v00
-rwx------    1 root     root            20 May  3 03:14 features.gz
-rwx------    1 root     root        407615 May  3 03:18 hfcndd.v00
-rwx------    1 root     root         17104 May  3 03:18 hid_hid.v00
-rwx------    1 root     root        518918 May  3 03:18 hiodrive.v00
-rwx------    1 root     root        164359 May  3 03:18 i40en.v00
-rwx------    1 root     root         52443 May  3 03:18 iavmd.v00
-rwx------    1 root     root        102754 May  3 03:18 igbn.v00
-rwx------    1 root     root        523433 May  3 03:17 ima_be2i.v00
-rwx------    1 root     root         33532 May  3 03:17 ima_csio.v00
-rwx------    1 root     root        366390 May  3 03:18 ima_qla4.v00
-rwx------    1 root     root        267718 May  3 03:21 imgdb.tgz
-rwx------    1 root     root         26391 May  3 03:18 intel_nv.v00
-rwx------    1 root     root       2256791 May  3 03:18 iomemory.v00
-rwx------    1 root     root         10012 May  3 03:18 ipmi_ipm.v00
-rwx------    1 root     root         24092 May  3 03:18 ipmi_ipm.v01
-rwx------    1 root     root         28837 May  3 03:18 ipmi_ipm.v02
-rwx------    1 root     root        151020 May  3 03:18 ixgben.v00
-rwx------    1 root     root            58 Jul 15 08:36 jumpstrt.gz
-rwx------    1 root     root       3853650 May  3 03:14 k.b00
-rwx------    1 root     root        591276 May  3 03:18 lpfc.v00
-rwx------    1 root     root         88975 May  3 03:17 lsi_mr3.v00
-rwx------    1 root     root        133877 May  3 03:17 lsi_msgp.v00
-rwx------    1 root     root        125413 May  3 03:18 lsi_msgp.v01
-rwx------    1 root     root        128945 May  3 03:18 lsi_msgp.v02
-rwx------    1 root     root         55139 May  3 03:18 lsu_hp_h.v00
-rwx------    1 root     root         97901 May  3 03:18 lsu_lsi_.v00
-rwx------    1 root     root        182053 May  3 03:18 lsu_lsi_.v01
-rwx------    1 root     root         95786 May  3 03:18 lsu_lsi_.v02
-rwx------    1 root     root        206602 May  3 03:18 lsu_lsi_.v03
-rwx------    1 root     root          5548 May  3 03:18 misc_cni.v00
-rwx------    1 root     root        363370 May  3 03:18 misc_dri.v00
-rwx------    1 root     root         70995 May  3 03:18 mtip32xx.v00
-rwx------    1 root     root         56045 May  3 03:18 mx_nvme.v00
-rwx------    1 root     root        182275 May  3 03:18 native_m.v00
-rwx------    1 root     root        224292 May  3 03:18 native_s.v00
-rwx------    1 root     root        176852 May  3 03:18 ne1000.v00
-rwx------    1 root     root         56575 May  3 03:18 nenic.v00
-rwx------    1 root     root       1146794 May  3 03:18 net_bna.v00
-rwx------    1 root     root        181347 May  3 03:18 net_bnx2.v00
-rwx------    1 root     root       1053368 May  3 03:18 net_bnx2.v01
-rwx------    1 root     root          6884 May  3 03:18 net_cdc_.v00
-rwx------    1 root     root         49589 May  3 03:18 net_cnic.v00
-rwx------    1 root     root        988015 May  3 03:17 net_cxgb.v00
-rwx------    1 root     root         95790 May  3 03:18 net_e100.v00
-rwx------    1 root     root        110400 May  3 03:18 net_e100.v01
-rwx------    1 root     root         53057 May  3 03:17 net_enic.v00
-rwx------    1 root     root         23926 May  3 03:18 net_fcoe.v00
-rwx------    1 root     root         38077 May  3 03:18 net_forc.v00
-rwx------    1 root     root        151241 May  3 03:18 net_i40e.v00
-rwx------    1 root     root         40577 May  3 03:18 net_ibma.v00
-rwx------    1 root     root        113714 May  3 03:18 net_igb.v00
-rwx------    1 root     root        187375 May  3 03:18 net_ixgb.v00
-rwx------    1 root     root         24719 May  3 03:18 net_libf.v00
-rwx------    1 root     root        180289 May  3 03:18 net_mlx4.v00
-rwx------    1 root     root         76520 May  3 03:18 net_mlx4.v01
-rwx------    1 root     root         69338 May  3 03:18 net_mlx_.v00
-rwx------    1 root     root       1379567 May  3 03:18 net_nx_n.v00
-rwx------    1 root     root        285910 May  3 03:18 net_qlcn.v00
-rwx------    1 root     root         89848 May  3 03:18 net_qlge.v00
-rwx------    1 root     root        101194 May  3 03:18 net_sgb.v00
-rwx------    1 root     root        169163 May  3 03:18 net_sxgb.v00
-rwx------    1 root     root        112994 May  3 03:18 net_tg3.v00
-rwx------    1 root     root         17221 May  3 03:18 net_usbn.v00
-rwx------    1 root     root         31432 May  3 03:18 net_vmxn.v00
-rwx------    1 root     root        141434 May  3 03:18 nhpsa.v00
-rwx------    1 root     root        140652 May  3 03:18 nmlx4_co.v00
-rwx------    1 root     root        142630 May  3 03:18 nmlx4_en.v00
-rwx------    1 root     root         64917 May  3 03:18 nmlx4_rd.v00
-rwx------    1 root     root        341393 May  3 03:18 nmlx5_co.v00
-rwx------    1 root     root         76275 May  3 03:18 nmlx5_rd.v00
-rwx------    1 root     root        315508 May  3 03:18 nqlcnic.v00
-rwx------    1 root     root         31854 May  3 03:18 ntg3.v00
-rwx------    1 root     root         78187 May  3 03:18 nvme.v00
-rwx------    1 root     root         43851 May  3 03:18 nvmxnet3.v00
-rwx------    1 root     root         20463 May  3 03:18 ohci_usb.v00
-rwx------    1 root     root         11290 May  3 03:21 onetime.tgz
-rwx------    1 root     root         18254 May  3 03:18 pvscsi.v00
-rwx------    1 root     root         81784 May  3 03:18 qcnic.v00
-rwx------    1 root     root       1079014 May  3 03:18 qedentv.v00
-rwx------    1 root     root        925324 May  3 03:18 qedrntv.v00
-rwx------    1 root     root        759260 May  3 03:18 qfle3.v00
-rwx------    1 root     root        299597 May  3 03:18 qfle3f.v00
-rwx------    1 root     root         99334 May  3 03:18 qfle3i.v00
-rwx------    1 root     root        199682 May  3 03:18 qflge.v00
-rwx------    1 root     root        766200 May  3 03:18 qlnative.v00
-rwx------    1 root     root        215871 May  3 03:18 rste.v00
-rwx------    1 root     root      69838762 May  3 03:17 s.v00
-rwx------    1 root     root      54072497 May  3 03:18 sas_vgc.v00
-rwx------    1 root     root         20966 May  3 03:18 sata_ahc.v00
-rwx------    1 root     root         12225 May  3 03:18 sata_ata.v00
-rwx------    1 root     root         17727 May  3 03:18 sata_sat.v00
-rwx------    1 root     root         10931 May  3 03:18 sata_sat.v01
-rwx------    1 root     root         11510 May  3 03:18 sata_sat.v02
-rwx------    1 root     root          8280 May  3 03:18 sata_sat.v03
-rwx------    1 root     root          6562 May  3 03:18 sata_sat.v04
-rwx------    1 root     root      43673181 May  3 03:15 sb.v00
-rwx------    1 root     root         89268 May  3 03:17 scsi_aac.v00
-rwx------    1 root     root       1453935 May  3 03:17 scsi_adp.v00
-rwx------    1 root     root        150556 May  3 03:18 scsi_adp.v01
-rwx------    1 root     root         99849 May  3 03:18 scsi_aic.v00
-rwx------    1 root     root         28269 May  3 03:19 scsi_arc.v00
-rwx------    1 root     root        254055 May  3 03:17 scsi_be2.v00
-rwx------    1 root     root       1257497 May  3 03:18 scsi_bfa.v00
-rwx------    1 root     root         98057 May  3 03:18 scsi_bnx.v00
-rwx------    1 root     root         81431 May  3 03:18 scsi_bnx.v01
-rwx------    1 root     root        178906 May  3 03:17 scsi_cel.v00
-rwx------    1 root     root        161733 May  3 03:17 scsi_cel.v01
-rwx------    1 root     root       1045388 May  3 03:17 scsi_csi.v00
-rwx------    1 root     root        437563 May  3 03:17 scsi_esa.v00
-rwx------    1 root     root        675068 May  3 03:17 scsi_esa.v01
-rwx------    1 root     root        698719 May  3 03:17 scsi_esa.v02
-rwx------    1 root     root         72681 May  3 03:17 scsi_fni.v00
-rwx------    1 root     root        470662 May  3 03:18 scsi_hfc.v00
-rwx------    1 root     root         83776 May  3 03:18 scsi_hps.v00
-rwx------    1 root     root         32968 May  3 03:18 scsi_ips.v00
-rwx------    1 root     root         11556 May  3 03:18 scsi_isc.v00
-rwx------    1 root     root         65336 May  3 03:18 scsi_lib.v00
-rwx------    1 root     root         65269 May  3 03:17 scsi_meg.v00
-rwx------    1 root     root         28630 May  3 03:18 scsi_meg.v01
-rwx------    1 root     root         27188 May  3 03:18 scsi_meg.v02
-rwx------    1 root     root        121039 May  3 03:17 scsi_mpt.v00
-rwx------    1 root     root        134562 May  3 03:18 scsi_mpt.v01
-rwx------    1 root     root        149386 May  3 03:18 scsi_mpt.v02
-rwx------    1 root     root        126785 May  3 03:18 scsi_mpt.v03
-rwx------    1 root     root         48921 May  3 03:18 scsi_ocz.v00
-rwx------    1 root     root        147904 May  3 03:18 scsi_qla.v00
-rwx------    1 root     root        396687 May  3 03:18 scsi_roc.v00
-rwx------    1 root     root         72719 May  3 03:17 scsi_sni.v00
-rwx------    1 root     root         35441 May  3 03:18 scsi_ste.v00
-rwx------    1 root     root          2483 May  3 03:18 shim_isc.v00
-rwx------    1 root     root          2483 May  3 03:18 shim_isc.v01
-rwx------    1 root     root          6932 May  3 03:18 shim_lib.v00
-rwx------    1 root     root          6932 May  3 03:18 shim_lib.v01
-rwx------    1 root     root          3443 May  3 03:18 shim_lib.v02
-rwx------    1 root     root          3442 May  3 03:18 shim_lib.v03
-rwx------    1 root     root          1599 May  3 03:18 shim_lib.v04
-rwx------    1 root     root          1600 May  3 03:18 shim_lib.v05
-rwx------    1 root     root         33558 May  3 03:18 shim_vmk.v00
-rwx------    1 root     root         35205 May  3 03:18 shim_vmk.v01
-rwx------    1 root     root         35748 May  3 03:18 shim_vmk.v02
-rwx------    1 root     root         71491 May  3 03:18 smartpqi.v00
-rwx------    1 root     root         78345 May  3 03:18 smgnvme.v00
-rwx------    1 root     root         19687 Jul 16 19:01 state.tgz
-rwx------    1 root     root         45075 May  3 03:14 tboot.b00
-rwx------    1 root     root          5541 May  3 03:14 uc_amd.b00
-rwx------    1 root     root       1253061 May  3 03:14 uc_intel.b00
-rwx------    1 root     root         20714 May  3 03:18 uhci_usb.v00
-rwx------    1 root     root         41259 May  3 03:18 usb_stor.v00
-rwx------    1 root     root         99490 May  3 03:18 usbcore_.v00
-rwx------    1 root     root        514725 May  3 03:14 user.b00
-rwx------    1 root     root            21 May  3 03:46 useropts.gz
-rwx------    1 root     root        222353 May  3 03:18 vmkata.v00
-rwx------    1 root     root         12436 May  3 03:18 vmkplexe.v00
-rwx------    1 root     root        424716 May  3 03:18 vmkusb.v00
-rwx------    1 root     root         77673 May  3 03:18 vmw_ahci.v00
-rwx------    1 root     root         42748 May  3 03:18 vmware_e.v00
-rwx------    1 root     root      11935240 May  3 03:19 vsan.v00
-rwx------    1 root     root        439194 May  3 03:19 vsanheal.v00
-rwx------    1 root     root       2883313 May  3 03:19 vsanmgmt.v00
-rwx------    1 root     root       2022420 May  3 03:18 weaselin.t00
-rwx------    1 root     root         75123 May  3 03:18 xhci_xhc.v00
-rwx------    1 root     root       1402646 May  3 03:19 xorg.v00
 
Last edited:

Zack Hehmann

Member
Feb 6, 2016
67
5
8
I also came across this site. I used the ISO from VMware directly and did not use a vendor ISO. Because I'm so close the the PSOD timser, I might want and see if time passes with out the kernel panic happening. The vib removal process requires a host reboot, so I'm going to wait until 2 days is up.

Here is the output for the loaded vibs.

Code:
[root@localhost:/vmfs/volumes/18c763e4-898c5a70-9366-b728a4f82b4f] esxcli software vib get | egrep Summary:
   Summary: celerity8fc: scsi driver for VMware ESX
   Summary: esas2hba: scsi driver for VMware ESX
   Summary: esas4hba: scsi driver for VMware ESX
   Summary: celerity16fc driver
   Summary: esashba4 driver
   Summary: aacraid: scsi driver for VMware ESX
   Summary: adp80xx: scsi driver for VMware ESX
   Summary: Avago Native MegaRAID SAS
   Summary: Avago Native MegaRAID SAS
   Summary: Avago (LSI) Native 12Gbps SAS/PCIe MPT Driver
   Summary: megaraid_sas: scsi driver for VMware ESX
   Summary: mpt3sas: scsi driver for VMware ESX
   Summary: Network driver for Broadcom NetXtreme-E Ethernet Controller
   Summary: Broadcom NetXtreme-E ROCE Driver
   Summary: enic: net driver for VMware ESX
   Summary: fnic: scsi driver for VMware ESX
   Summary: snic: scsi driver for VMware ESX
   Summary: Chelsio Communications T5/T6 Native Network Driver
   Summary: csiostor_ima: IMA plugin for VMware ESX
   Summary: cxgb4: net driver for VMware ESX
   Summary: csiostor: scsi driver for VMware ESX
   Summary: beima: IMA plugin for VMware ESX
   Summary: be2iscsi: scsi driver for VMware ESX
   Summary: vgc: FlashMAX driver for VMware ESXi
   Summary: hpsa: scsi driver for VMware ESX
   Summary: Hitachi FC Driver
   Summary: hfcldd: scsi driver for VMware ESX
   Summary: hio: block driver for VMware ESX
   Summary: Non-Volatile memory controller driver
   Summary: ibma_driver: net driver for VMware ESX
   Summary: rockraid: scsi driver for VMware ESX
   Summary: Network driver for Intel(R) X710/XL710/X722 Adapters
   Summary: Intel NVME Driver with VMD Technology
   Summary: Network driver for Intel(R) 1 Gigabit Server NIC
   Summary: The Intel NVM-Express driver
   Summary: Network driver for Intel(R) 10 Gigabit NIC
   Summary: i40e: net driver for VMware ESX
   Summary: igb: net driver for VMware ESX
   Summary: ixgbe: net driver for VMware ESX
   Summary: Mangstor NVMe driver
   Summary: Mellanox Technologies ConnectX4 Native Core Driver
   Summary: Mellanox Technologies ConnectX-4 Native RDMA Driver
   Summary: mlx_compat: net driver for VMware ESX
   Summary: mlx4_core: net driver for VMware ESX
   Summary: mlx4_en: net driver for VMware ESX
   Summary: HPSA native driver
   Summary: ocz10xx: scsi driver for VMware ESX
   Summary: Network driver for QLogic 1G/10G Ethernet Controller
   Summary: QLogic E3 Common Interface driver
   Summary: Network driver for QLogic FastLinQ QL45xxx Ethernet Controller
   Summary: RoCE driver for QLogic FastLinQ QL45xxx Ethernet Controller
   Summary: Network driver for QLogic E3 Controller
   Summary: QLogic E3 Native FC driver
   Summary: Native E3 iSCSI Driver
   Summary: QLogic IMA plugin for VMware ESX, ESXi
   Summary: cnic_register: misc driver for VMware ESX
   Summary: bna: net driver for VMware ESX
   Summary: bnx2: net driver for VMware ESX
   Summary: bnx2x: net driver for VMware ESX
   Summary: cnic: net driver for VMware ESX
   Summary: nx_nic: net driver for VMware ESX
   Summary: qlcnic: net driver for VMware ESX
   Summary: qlge: net driver for VMware ESX
   Summary: Qlogic Native FC driver
   Summary: bfa: scsi driver for VMware ESX
   Summary: bnx2fc: scsi driver for VMware ESX
   Summary: bnx2i: scsi driver for VMware ESX
   Summary: qla4xxx: scsi driver for VMware ESX
   Summary: Non-Volatile memory controller driver
   Summary: Fusion ioMemory storage driver
   Summary: Native Shannon Driver
   Summary: sgb: net driver for VMware ESX
   Summary: sxgbe: net driver for VMware ESX
   Summary: stec_s1120: scsi driver for VMware ESX
   Summary: libata_92: ata driver for VMware ESX
   Summary: pata_amd: ata driver for VMware ESX
   Summary: pata_atiixp: ata driver for VMware ESX
   Summary: pata_cmd64x: ata driver for VMware ESX
   Summary: pata_hpt3x2n: ata driver for VMware ESX
   Summary: pata_pdc2027x: ata driver for VMware ESX
   Summary: pata_serverworks: ata driver for VMware ESX
   Summary: pata_sil680: ata driver for VMware ESX
   Summary: pata_via: ata driver for VMware ESX
   Summary: cciss: block driver for VMware ESX
   Summary: Updates the ESX 6.5.0 brcmfcoe
   Summary: random: char driver for VMware ESX
   Summary: ehci-hcd: ehci driver for VMware ESX
   Summary: net driver for VMware ESX
   Summary: hid: hid driver for VMware ESX
   Summary: Updates the ESX 6.5.0 ipmi-ipmi-devintf
   Summary: Updates the ESX 6.5.0 ipmi-ipmi-msghandler
   Summary: ipmi_si_drv: ipmi driver for VMware ESX
   Summary: Updates the ESX 6.5.0 lpfc
   Summary: Updates the ESX 6.5.0 lsi-msgpt2
   Summary: Updates the ESX 6.5.0 lsi-msgpt3
   Summary: Updates the ESX 6.5.0 misc-drivers
   Summary: P32x/P42x PCIe SSD
   Summary: Updates the ESX 6.5.0 ne1000
   Summary: Cisco VIC Native driver for VMware ESX
   Summary: cdc_ether: net driver for VMware ESX
   Summary: e1000: net driver for VMware ESX
   Summary: e1000e: net driver for VMware ESX
   Summary: fcoe: net driver for VMware ESX
   Summary: forcedeth: net driver for VMware ESX
   Summary: libfcoe_92: net driver for VMware ESX
   Summary: tg3: net driver for VMware ESX
   Summary: usbnet: net driver for VMware ESX
   Summary: vmxnet3: net driver for VMware ESX
   Summary: Mellanox Technologies ConnectX Native Core Driver
   Summary: Mellanox Technologies ConnectX Native Ethernet Driver
   Summary: Mellanox Technologies ConnectX Native RDMA Driver
   Summary: Broadcom NetXtreme I ESX VMKAPI ethernet driver
   Summary: Updates the ESX 6.5.0 nvme
   Summary: Network driver for VMware vmxnet3 Virtual Ethernet Controller
   Summary: usb-ohci: ohci driver for VMware ESX
   Summary: PVSCSI Driver
   Summary: Network driver for QLogic NetXtreme II PCI/PCIe Gigabit Ethernet Adapters
   Summary: ahci: sata driver for VMware ESX
   Summary: ata_piix: sata driver for VMware ESX
   Summary: sata_nv: sata driver for VMware ESX
   Summary: sata_promise: sata driver for VMware ESX
   Summary: sata_sil24: sata driver for VMware ESX
   Summary: sata_sil: sata driver for VMware ESX
   Summary: sata_svw: sata driver for VMware ESX
   Summary: adp94xx: scsi driver for VMware ESX
   Summary: aic79xx: scsi driver for VMware ESX
   Summary: ips: scsi driver for VMware ESX
   Summary: iscsi_linux_92: scsi driver for VMware ESX
   Summary: libfc_92: scsi driver for VMware ESX
   Summary: megaraid_mbox: scsi driver for VMware ESX
   Summary: megaraid2: scsi driver for VMware ESX
   Summary: mpt2sas: scsi driver for VMware ESX
   Summary: mptsas: scsi driver for VMware ESX
   Summary: mptspi: scsi driver for VMware ESX
   Summary: iscsi_linux_9_2_1_0: shim driver for VMware ESX
   Summary: iscsi_linux_9_2_2_0: shim driver for VMware ESX
   Summary: libata_9_2_1_0: shim driver for VMware ESX
   Summary: libata_9_2_2_0: shim driver for VMware ESX
   Summary: libfc_9_2_1_0: shim driver for VMware ESX
   Summary: libfc_9_2_2_0: shim driver for VMware ESX
   Summary: libfcoe_9_2_1_0: shim driver for VMware ESX
   Summary: libfcoe_9_2_2_0: shim driver for VMware ESX
   Summary: vmklinux_9_2_1_0: shim driver for VMware ESX
   Summary: vmklinux_9_2_2_0: shim driver for VMware ESX
   Summary: vmklinux_9_2_3_0: shim driver for VMware ESX
   Summary: Updates the ESX 6.5.0 smartpqi
   Summary: usb-uhci: uhci driver for VMware ESX
   Summary: usb-storage: usb-storage driver for VMware ESX
   Summary: Updates the ESX 6.5.0 usbcore-usb
   Summary: ATA Driver
   Summary: vmkplexer: vmkplexer driver for VMware ESX
   Summary: Updates the ESX 6.5.0 vmkusb
   Summary: Updates the ESX 6.5.0 vmw-ahci
   Summary: xhci: xhci driver for VMware ESX
   Summary: CPU microcode updates
   Summary: esxcli plugin for Emulex elxnet driver
   Summary: Updates the ESX 6.5.0 esx-base
   Summary: dvfilter-generic-fastpath module
   Summary: Updates the ESX 6.5.0 esx-tboot
   Summary: VMware Host Client
   Summary: Updates the ESX 6.5.0 esx-xserver
   Summary: Updates the ESX 6.5.0 lsu-hp-hpsa-plugin
   Summary: LSI MR3 LSU Management Plugin
   Summary: LSI MSGPT3 LSU Management Plugin
   Summary: LSI MegaRAID SAS LSU Management Plugin
   Summary: LSI MPT2SAS LSU Management Plugin
   Summary: Miscellaneous drivers for ESX
   Summary: rste: SCU SAS/SATA for VMware ESX
   Summary: Updates the ESX 6.5.0 vmware-esx-esxcli-nvme-plugin
   Summary: Updates the ESX 6.5.0 vsan
   Summary: ESXi VSAN Health Service
   Summary: ESXi tools
   Summary: arcmsr: scsi driver for VMware ESX
 

Rand__

Well-Known Member
Mar 6, 2014
5,162
1,070
113
Whenever i had space issues it was either missing swap space or a weird bug (once).
I moved away from USB sticks for ESXi ages ago since i had nothing but trouble with them (when used with vcenter)
 

dswartz

Active Member
Jul 14, 2011
481
47
28
I went with 64 or 128GB SSD, since I am currently playing with hyper-converged storage, so I need a reasonable amount of space on a local datastore for the appliance.
 

Zack Hehmann

Member
Feb 6, 2016
67
5
8
Host is still up and running past the time it was getting before. Unfortunately there isn't a way to determine what the real cause is since I listed the three things that were changed.

I'm trying to figure out what to do next. Should I delete some unused vibs?

I don't have an easy way to change to a ssd/sata hdd in this host as all of the drive bays are occupied and in use. I may move to a different storage device for ESXi in the future, but for a short term thing, I don't think it's possible. I had plans to migrate to a Supermicro board instead, so this may be another reason to do it sooner than later.
 

Zack Hehmann

Member
Feb 6, 2016
67
5
8
Let's say I move to a bigger USB drive, would this be a good method to backup the settings so I can be back to where I was with the smaller flashdrive?

I'm certain I have used this before, but I'm not sure what you guys use or have done.

I thought that I read that increasing the local disk size where ESXi is installed, would make a difference on the size of the /bootblock directory?
 
Last edited: