Fresh ESXi 6.7 install; Error 28 No space left issue during latest update

nthu9280 · Feb 18, 2019

I just did a fresh install of ESXi 6.7. When I try to update it to the latest profile I get the following error. The install is on a 32GB mSATA device.

Code:

esxcli software profile update -p ESXi-6.7.0-20190104001-standard \
>-d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
 [OSError]
 [Errno 28] No space left on device
 Please refer to the log file for more details.

Code:

  File: "/"
    ID: 100000000 Namelen: 127     Type: visorfs
Block size: 4096
Blocks: Total: 568868     Free: 371228     Available: 371228
Inodes: Total: 655360     Free: 648812

Ptr Blocks shows max/free as 0/0. Is this the contributing factor?

Code:

vmkfstools -Ph -v 10 /vmfs/volumes/5c69e33b-ca55bcf6-dcee-ac1f6b82cd77
VMFS-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
File system label (if any): datastore1
Mode: public
Capacity 22.2 GB, 19.8 GB available, file block size 1 MB, max supported file size 64 TB
Volume Creation Time: Sun Feb 17 22:42:03 2019
Files (max/free): 16384/16374
Ptr Blocks (max/free): 0/0
Sub Blocks (max/free): 16384/16383
Secondary Ptr Blocks (max/free): 256/255
File Blocks (overcommit/used/overcommit %): 0/2463/0
Ptr Blocks  (overcommit/used/overcommit %): 0/0/0
Sub Blocks  (overcommit/used/overcommit %): 0/1/0
Large File Blocks (total/used/file block clusters): 45/0/45
Volume Metadata size: 1502584832
Disk Block Size: 512/512/0
UUID: 5c69e33b-ca55bcf6-dcee-ac1f6b82cd77
Logical device: 5c69e335-f12283e0-d913-ac1f6b82cd77
Partitions spanned (on "lvm"):
        t10.ATA_____LITEONIT_LMT2D32L3M______________________TW04NG44550852A25323:3
Is Native Snapshot Capable: NO
OBJLIB-LIB: ObjLib cleanup done.
WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

I was able to successfully update the vibs even though the software profile update fails. Shouldn't the vib update fail too if the above was the issue?

Code:

esxcli software vib update  \
> -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

- Checked and the Swap enabled
- tried the ESXi-6.7.0-20190104001-no-tools
- Did not see any large log files etc.

Code:

No large log files etc.
find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -lh '{}' \;

Appreciate any help I can get on this. I checked out most of the troubleshooting tips that I could find on the interwebz...

Rand__ · Feb 18, 2019

From my personal logs...

[Errno 28] No space left on device
Go to Host > Manage > System > Swap and activate swap on our datastore vmfs.

O/c I didn't discover it but was plagued by it so often that i saved it

Edit:
Oh I see you have swap enabled? Now thats weird then.

Edit2 - And to provide some thing useful, here is a comparison from a 32GB drive

Code:

 vmkfstools -Ph -v 10 /vmfs/volumes/Boot/
VMFS-5.81 (Raw Major Version: 14) file system spanning 1 partitions.
File system label (if any): Boot
Mode: public
Capacity 22.2 GB, 17.7 GB available, file block size 1 MB, max supported file size 62.9 TB
Volume Creation Time: Sun Jul 16 14:30:02 2017
Files (max/free): 109068/109058
Ptr Blocks (max/free): 64512/64492
Sub Blocks (max/free): 32000/31999
Secondary Ptr Blocks (max/free): 256/256
File Blocks (overcommit/used/overcommit %): 0/4697/0
Ptr Blocks  (overcommit/used/overcommit %): 0/20/0
Sub Blocks  (overcommit/used/overcommit %): 0/1/0
Volume Metadata size: 760766464
Disk Block Size: 512/512/0
UUID: 596b786a-749efb64-5b1b-0025905db545
Logical device: 596b7868-b921332c-60d6-0025905db545
Partitions spanned (on "lvm"):
        t10.ATA_____SATA_SSD________________________________67F407560E2400150722:3
Is Native Snapshot Capable: YES
OBJLIB-LIB: ObjLib cleanup done.
WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

Looks like your pointers are used up, but why...

Can you upgrade to a december build?

nthu9280 · Feb 18, 2019

There was a November build and that also failed. I don't think I saw a December build.

Swap was disabled on mine too. I enabled it and selected datastore as was suggested in one of the check lists

The "Ptr Blocks" is perplexing. It's clean install and I just took the defaults. I ran the suggested checks that in various KBs and google searches on this topic.

One thing I noticed is that my FS is VMFS-6.82 (Raw Major Version: 24). Guess I'll reinistall and try later this evening. Like noted earlier, vib update succeeded I can check the vib versions but they should be current. Just the profile still shows the 201810*.

Rand__ · Feb 18, 2019

Well my install is old, that will explain the difference

Just realized I had one from my HA debugging attempts as well

Code:

 vmkfstools -Ph -v 10 /vmfs/volumes/datastore1\ \(1\)/
VMFS-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
File system label (if any): datastore1 (1)
Mode: public
Capacity 22.2 GB, 20.8 GB available, file block size 1 MB, max supported file size 64 TB
Volume Creation Time: Mon Feb 11 21:02:14 2019
Files (max/free): 16384/16375
Ptr Blocks (max/free): 0/0
Sub Blocks (max/free): 16384/16384
Secondary Ptr Blocks (max/free): 256/255
File Blocks (overcommit/used/overcommit %): 0/1439/0
Ptr Blocks  (overcommit/used/overcommit %): 0/0/0
Sub Blocks  (overcommit/used/overcommit %): 0/0/0
Large File Blocks (total/used/file block clusters): 45/0/45
Volume Metadata size: 1502584832
Disk Block Size: 512/512/0
UUID: 5c61e2d6-49b1b1ac-9acc-248a07a0a9c8
Logical device: 5c61e2d5-efb3694c-89fd-248a07a0a9c8
Partitions spanned (on "lvm"):
        t10.ATA_____SATA_SSD________________________________67F407560C2400150309:3
Is Native Snapshot Capable: NO
OBJLIB-LIB: ObjLib cleanup done.
WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

That one also has 0 PTRs... it upgraded fine to the latest but also has weird behaviour

Same box, different installation disk (960GB), same 0 PTRs - so dont think its relevant on VMFS6

Code:

vmkfstools -Ph -v 10 /vmfs/volumes/datastore1\ \(2\)/
VMFS-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
File system label (if any): datastore1 (2)
Mode: public
Capacity 886.8 GB, 885.3 GB available, file block size 1 MB, max supported file size 64 TB
Volume Creation Time: Mon Feb 11 22:21:36 2019
Files (max/free): 16384/16375
Ptr Blocks (max/free): 0/0
Sub Blocks (max/free): 16384/16384
Secondary Ptr Blocks (max/free): 256/255
File Blocks (overcommit/used/overcommit %): 0/1453/0
Ptr Blocks  (overcommit/used/overcommit %): 0/0/0
Sub Blocks  (overcommit/used/overcommit %): 0/0/0
Large File Blocks (total/used/file block clusters): 1774/0/238
Volume Metadata size: 1517600768
Disk Block Size: 512/512/0
UUID: 5c61f570-0f00910c-75ab-248a07a0a9c8
Logical device: 5c61f56f-72a6976c-70c4-248a07a0a9c8
Partitions spanned (on "lvm"):
        t10.ATA_____SAMSUNG_MZ7KM960HAHP2D00005______________S2HTNX0HB00039______:3
Is Native Snapshot Capable: NO
OBJLIB-LIB: ObjLib cleanup done.
WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

nthu9280 · Feb 18, 2019

For now, i just manually downloaded the vib file, uploaded it to the datastore, and ran a VIB update.
esxcli software profile update -p ESXi-6.7.0-20190104001-no-tools -d <depot> is still failing.
-- need to figure out how to increase the /tmp on visorfs. It's 256MB limit I think is causing the update to fail.

Code:

esxcli system visorfs ramdisk list
Ramdisk Name  System  Include in Coredumps   Reserved      Maximum      Used  Peak Used   Free  Reserved Free  Maximum Inodes  Allocated Inodes  Used Inodes  Mount Point
------------  ------  --------------------  ---------  -----------  --------  ---------  -----  -------------  --------------  ----------------  -----------  ---------------------------
root            true                  true  32768 KiB    32768 KiB  2296 KiB   2296 KiB   92 %           92 %           16928              5248         5230  /
etc             true                  true  28672 KiB    28672 KiB   172 KiB    200 KiB   99 %           99 %            4096              1024          631  /etc
opt             true                  true      0 KiB    32768 KiB     0 KiB      0 KiB  100 %            0 %            8192              1024            7  /opt
var             true                  true   5120 KiB    49152 KiB   360 KiB    376 KiB   99 %           92 %            8192               672          665  /var
tmp            false                 false   2048 KiB   262144 KiB    12 KiB   1296 KiB   99 %           99 %            8192               256            5  /tmp
iofilters      false                 false      0 KiB    32768 KiB     0 KiB      0 KiB  100 %            0 %           10240                32            1  /var/run/iofilters
shm            false                 false      0 KiB  1048576 KiB     0 KiB      0 KiB  100 %            0 %             512                32            1  /var/run/shm
hostdstats     false                 false      0 KiB   160768 KiB  1788 KiB   1788 KiB   98 %            0 %            8192                32            4  /var/lib/vmware/hostd/stats

Funny thing is the build number shows the Jan-2019 version but the software profile shows Oct-2018

VMware Knowledge Base

Rand__ · Feb 19, 2019

Ouch - just realized I wiped three "productive" drives with copying your command blindly :O

nthu9280 · Feb 19, 2019

Ouch. Sorry to hear that.

On the same vein, imagine what I did couple of weeks ago. I was cleaning up old remnants on my Ubuntu 18.04 linux . Just up arrow change the last digit and run both the commands. Toward the end did 4.4.0-4*. I think you can guess the result. Fortunately, I was able to recover quickly.

Code:

for i in `find /lib/ -name "4.4.0-2*"` ; do echo $i ; done
for i in `find /lib/ -name "4.4.0-2*"` ; do rm -r $i ; done

Rand__ · May 12, 2019

So have you ever resolved that issue?
I think I just encountered it too ...

BoredSysadmin · May 12, 2019

check if you /bootbank folder points to /tmp.
If it does the fix for that is : VMware Knowledge Base

another option would be to find boot.cfg. You should find two. Find one with newer build file (cat it)
add devStabilityCount=10. save and reboot

Rand__ · May 12, 2019

thanks, but didnt help. Bootbank was fine, upgrade still fails :/

nthu9280 · May 12, 2019

I gave up on this. Tried few suggestions found on the interwebs but none of them worked. I should have opened an incident while eval was still active.

You can manually download the VIB and update but normal process kept failing. I even tried "no-tools" profile without success.

Rand__ · May 13, 2019

One can open Tickets during eval? For all products?
Totally news to me

i386 · May 13, 2019

I had the error 28 on a recent installtion too, swap was alaready enabled.

"Fixed" it by using

Code:

esxcli system module set -m=vmkusb -e=FALSE

before the installtion and

Code:

esxcli system module set -m=vmkusb -e=TRUE

after installtion.

Not sure if it can help you guys.

Rand__ · May 13, 2019

That did not help either, but i also decided not to spend any more time on this and to do the installation workaround.

There are a couple of links available when searching for "No space left on device vibs = VMware_locker_tools-light" , eg
No Space Left On Device vmware_locker_tools error when upgrading ESXI 6.7

that worked

Edit - this does not seem to be exactly the same issue as the OP, his was not related to a particular vib it seems

nthu9280 · May 13, 2019

Rand__ said:
One can open Tickets during eval? For all products?
Totally news to me

I opened a SR during eval period back in the ESXi 5.x days I think. Not sure if they still allow that.

Search

Fresh ESXi 6.7 install; Error 28 No space left issue during latest update

nthu9280

Well-Known Member

Rand__

Well-Known Member

nthu9280

Well-Known Member

Rand__

Well-Known Member

nthu9280

Well-Known Member

Rand__

Well-Known Member

nthu9280

Well-Known Member

Rand__

Well-Known Member

BoredSysadmin

Not affiliated with Maxell

Rand__

Well-Known Member

nthu9280

Well-Known Member

Rand__

Well-Known Member

i386

Well-Known Member

Rand__

Well-Known Member

nthu9280

Well-Known Member