Thank you both for providing great resources for me to look into. Looks like I've got some HW to do. I'll check back in once I have some time to dive into this and get some useful data to go off of.
Slow power on of VMs...
- Thread starter JimPhreak
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Everyone's a n00b at first...! 
Reading the docs is a great way to start but the only way you ever really get to geek out on a product is when you get weird "this shouldn't be happening" problems like this - something that vmware gets more than its fair share of simply because of its fairly mind-boggling complexity.
But yeah... this is interesting because you've pretty much got a KISS setup with some very fast (albeit fairly bleeding edge) IO so I'm kinda in the dark about why ESX would be waiting for 45s to bring a guest up. Actually, just to rule that out, have you got any other storage presented to this box that you can try moving a VM to to see if the issue occurs there?
Reading the docs is a great way to start but the only way you ever really get to geek out on a product is when you get weird "this shouldn't be happening" problems like this - something that vmware gets more than its fair share of simply because of its fairly mind-boggling complexity.But yeah... this is interesting because you've pretty much got a KISS setup with some very fast (albeit fairly bleeding edge) IO so I'm kinda in the dark about why ESX would be waiting for 45s to bring a guest up. Actually, just to rule that out, have you got any other storage presented to this box that you can try moving a VM to to see if the issue occurs there?
That was my first thought when I noticed this issue since I had to manually install the drivers for the SATA AHCI controller in order to get the Samsung SM951 M.2 drive to show up as a device in VMware. However I've already tried moving VM's to the Intel 730 480GB SSD I've got as a separate datastore and the issue still exists.Everyone's a n00b at first...!
But yeah... this is interesting because you've pretty much got a KISS setup with some very fast (albeit fairly bleeding edge) IO so I'm kinda in the dark about why ESX would be waiting for 45s to bring a guest up. Actually, just to rule that out, have you got any other storage presented to this box that you can try moving a VM to to see if the issue occurs there?
Couple thinks I just noticed in my vmkernal.log. First off the timestamps in all my logs are 6 hours ahead of my ESXi host which is configured with NTP. Can't get the log times to change no matter what I do with the ESXi host (disable NTP, change NTP servers, restart service, etc.)
But anyway, in the logs I'm seeing these messages every time I have a USB device passed through to a VM. They stop the moment I shut off that VM (this goes for my unRAID VM or a Windows VM).
2015-06-11T16:14:41.552Z cpu10:717671)<6>usb 2-1.2: Device is allocated for USB passthrough use; not available for VMkernel use
2015-06-11T16:14:41.668Z cpu0:717671)<6>usb 2-1.2: reset high speed USB device number 10 using ehci_hcd
2015-06-11T16:14:42.284Z cpu7:717544)<6>usb 2-1.2: device is available for passthrough
Then I just noticed this in the log (there are multiple of these relating to more than one SSD):
2015-06-11T16:20:00.049Z cpu1:32902)WARNING: LinScsi: SCSILinuxQueueCommand:1207: queuecommand failed with status = 0x1056 Unknown status vmhba0:0:0:0 (driver name: ahci) - Message repeated 1 time
2015-06-11T16:20:00.049Z cpu15:35120)ScsiDeviceIO: 2324: Cmd(0x412e85e7e080) 0x28, CmdSN 0x39c1c from world 32902 to dev "t10.ATA_____SATA_SSD________________________________96D707531A2400148195" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2015-06-11T16:20:00.467Z cpu2:32799)ScsiDeviceIO: 2324: Cmd(0x412e80812640) 0x85, CmdSN 0x4b7a from world 0 to dev "t10.ATA_____SATA_SSD________________________________96D707531A2400148195" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
Not sure what these mean or what they're affecting if anything
But anyway, in the logs I'm seeing these messages every time I have a USB device passed through to a VM. They stop the moment I shut off that VM (this goes for my unRAID VM or a Windows VM).
2015-06-11T16:14:41.552Z cpu10:717671)<6>usb 2-1.2: Device is allocated for USB passthrough use; not available for VMkernel use
2015-06-11T16:14:41.668Z cpu0:717671)<6>usb 2-1.2: reset high speed USB device number 10 using ehci_hcd
2015-06-11T16:14:42.284Z cpu7:717544)<6>usb 2-1.2: device is available for passthrough
Then I just noticed this in the log (there are multiple of these relating to more than one SSD):
2015-06-11T16:20:00.049Z cpu1:32902)WARNING: LinScsi: SCSILinuxQueueCommand:1207: queuecommand failed with status = 0x1056 Unknown status vmhba0:0:0:0 (driver name: ahci) - Message repeated 1 time
2015-06-11T16:20:00.049Z cpu15:35120)ScsiDeviceIO: 2324: Cmd(0x412e85e7e080) 0x28, CmdSN 0x39c1c from world 32902 to dev "t10.ATA_____SATA_SSD________________________________96D707531A2400148195" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2015-06-11T16:20:00.467Z cpu2:32799)ScsiDeviceIO: 2324: Cmd(0x412e80812640) 0x85, CmdSN 0x4b7a from world 0 to dev "t10.ATA_____SATA_SSD________________________________96D707531A2400148195" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
Not sure what these mean or what they're affecting if anything
Aha, sorry Jim I'd missed your post from thursday - and yes, as you rightly surmised those error messages would have given rise to freezes on your local datastores; even if you don't have owt plugged into the M1015 I suspect it would still cause freezes on the whole SCSI stack with a knock-on effect for AHCI gubbins which would explain the long power-up time whilst vmkernel waits to make sure the device is reachable. Glad you've got it sorted with a simple fix anyhoo.
Funny but glad it was an easy fix once...kinda odd that the issue did not manifest/surface to esxtop stats. Hmm color me perplexed.
Well none of the disks attached to my M1015 were in use by VMware. The only disks I setup datastores on were connected to my on board controllers.
So you moved them to the M1015 or just simply re-seated HBA ctrl and the onboard started playing nice/becomming responsive to VM pwr on operations.
All I did was reseat the PCIE card. When I first got the card I only had a standard PCI bracket for it not realizing the SuperMicro chassis was low profile. So I actually had the M1015 in the slot with no bracket. Once I got the low profile bracket I re-seated the M1015 and now everything is working.
Wow and cool...that's a SMH moment for sure...just what is the M1015 in there for then? Future expansion? AIO? Pile o' SSD's to form and all-flash-array...Do tell.
Attached to the M1015 are 4 x 8TB Seagate shingled drives + 1 x Intel 730 SSD that I'm passing through to my unRAID storage VM.