Can't shut down ESXi 5.5 host fully

Discussion in 'VMware, VirtualBox, Citrix' started by ZeroOne, Feb 5, 2014.

  1. ZeroOne

    ZeroOne Member

    Joined:
    Sep 13, 2013
    Messages:
    52
    Likes Received:
    6
    Flashed the ASMB7-iKVM back to 1.05. Seems the CPU issue is fixed. The system still cannot be shut down with this installed though. Same hang on shutdown. Almost not worth it, to leave it installed, once the system is up and configured.


    Update: 1.05 did not fix the issue. Removing the card entirely until a new update is available to test. Not sure if it's a system board bios, or the add on card firmware.
     
    #21
    Last edited: Feb 14, 2014
  2. jac

    jac Member

    Joined:
    Oct 21, 2012
    Messages:
    48
    Likes Received:
    0
    Is this happening right up to the latest 5.5b build?
     
    #22
  3. ZeroOne

    ZeroOne Member

    Joined:
    Sep 13, 2013
    Messages:
    52
    Likes Received:
    6
    Using this ESXi:
    2013-09-22 | 5.5.0 | 326M | .iso
     
    #23
  4. jingjing

    jingjing Banned

    Joined:
    Nov 23, 2013
    Messages:
    22
    Likes Received:
    0
    sounds like the drivers for the motherboard are incomplete as far as power management, does it show in vcenter your power load under advanced performance ?
     
    #24
  5. ZeroOne

    ZeroOne Member

    Joined:
    Sep 13, 2013
    Messages:
    52
    Likes Received:
    6
    The graph under the Performance tab, using the drop-down to select "power?" If so, not currently. It did before I removed the add-on card, though it showed like 300 watts, if I'm remembering correctly. Wall power was closer to 110w at that time, measured with an inline meter. After removing the add-on card, it now sits at 0w.
     
    #25
  6. 33_viper_33

    33_viper_33 Member

    Joined:
    Aug 3, 2013
    Messages:
    199
    Likes Received:
    2
    I've been eying this board since its release. Any luck with the shutdown issue? I see firmware version 1.08.2 is available forthe ASMB7-iKVM.
     
    #26
  7. Borsaid

    Borsaid New Member

    Joined:
    Aug 8, 2014
    Messages:
    2
    Likes Received:
    0
    I've run into the same problem as ZeroOne, but with the ASUS P9D-I board. The ASMB7-IKVM is onboard as well. Although I have not been able to replicate the CPU spike issue on the ASMB7-IKVM Firmware v 1.08, I definitely can not shutdown ESXi cleanly.

    I was hoping someone has had progress on this issue since February 2014.
     
    #27
  8. Borsaid

    Borsaid New Member

    Joined:
    Aug 8, 2014
    Messages:
    2
    Likes Received:
    0
    Scratch that. Walked into the office this morning and AM having that exact same CPU spike issue. Downgraded the IKVM firmware this morning and rebooted.
     
    #28
  9. mbau

    mbau New Member

    Joined:
    Aug 27, 2014
    Messages:
    2
    Likes Received:
    0
    Same problem here with ESXi 5.5, an ASUS Z9PA-D8 with a ASMB6 iKVM Controller.

    ESXi not power off the host.
     
    #29
  10. Daniel_GT

    Daniel_GT New Member

    Joined:
    Sep 25, 2014
    Messages:
    9
    Likes Received:
    0
    I'd like to chip in on this one. I'm using the ASMB6 with the 1.10 firmware, on a Z9PE-D8 WS board and ESXi 5.5 update 2.

    Can confirm that, out of the box so to speak, the host will not perform a graceful shutdown. It seems to send the signal which starts the process and stalls right at the end, spinning up the fans. The host can be turned off at this point by sending a "Power Off - Immediately" command from the web client though this is patchy. I also rolled back at least two driver versions which didn't help.

    I contacted ASUS support who referred me to their OS compatibility, which doesn't include ESXi - damn shame as most if not all of the other features work well. Considering that the last firmware version is a couple of years old I don't expect they'll be adding it anytime soon though I've found having a meaningful conversation with them quite difficult.

    I'm wondering if some fine tuning of board BIOS settings for the interface AND power management (ACPI is it?) might lead to a solution. I've also noted that within vCenter Server an IPMI setting can be set (user, password, IP, MAC) though fails to communicate properly. VMWare KB suggests it's the protocol version.

    I'd like to think that a solution can be found, feels to me that a fix should be possible? I'll update as I investigate further.
     
    #30
  11. mbau

    mbau New Member

    Joined:
    Aug 27, 2014
    Messages:
    2
    Likes Received:
    0
    I checked with VMWare support the shutdown problem and they confirm that the issue is the ASMB6.

    I also contacted the ASUS support, but there i not received a feedback till today. The funny thing is, is a server system TS700-X7 from ASUS on the compatibility guide from VMWare which also contains an ASMB6 and an ASUS Z9PA-D8 mainboard like i use.

    I wonder if Asus uses there a special firmware which is not public available?
     
    #31
  12. Ulrich Frey

    Ulrich Frey New Member

    Joined:
    Dec 29, 2014
    Messages:
    2
    Likes Received:
    1
    Hello,

    there is a new beta BIOS from ASUS with version number 0803 which fixes the shutdown problem.
     
    #32
  13. Daniel_GT

    Daniel_GT New Member

    Joined:
    Sep 25, 2014
    Messages:
    9
    Likes Received:
    0
    Thanks Ulrich, do you have a link you can share? Can't see anything on the official downloads page for either my mobo (see above) or the ASMB6?
     
    #33
  14. Ulrich Frey

    Ulrich Frey New Member

    Joined:
    Dec 29, 2014
    Messages:
    2
    Likes Received:
    1
    #34
    Daniel_GT likes this.
  15. ZeroOne

    ZeroOne Member

    Joined:
    Sep 13, 2013
    Messages:
    52
    Likes Received:
    6
    Actually made some headway on this and can shut down now. Turns out, there's a flaw in ESXi. Specifically, a driver for IPMI built in that locks one core of the CPU up. Stumbled on this, of all places, on an Amazon review of another product. For what it's worth, I'm also running the latest motherboard bios from 4/21/2014 and ASMB7 firmware 1.08.2 from 1/20/2014.

    Quoting the Amazon post:
    VMWARE ESXi STABILITY NOTE: ESXi has an issue with many motherboards that have IPMI sensors on board where a runaway process will consume most of the CPU timeslices even without any guest OS running. A workaround for this problem is to go into Configuration -> Software -> Advanced Settings -> VMkernel -> VMkernel.Boot.ipmiEnabled and uncheck the box. Unfortunately you lose the ability to monitor temperatures, fan speeds, etc. with VMware (vSphere Client among others) software. Fortunately you can go into the web-interface for the IPMI and see the same information.
     
    #35
    Daniel_GT and Chuckleb like this.
  16. ZeroOne

    ZeroOne Member

    Joined:
    Sep 13, 2013
    Messages:
    52
    Likes Received:
    6
    Just to reiterate, following the steps to disable the VMkernel.Boot.ipmiEnabled value does allow me to shut the host completely down as expected.

    This, combined with the CyberPower appliance has things working perfectly. 60 minutes after the power goes out, the VMs shut down in succession followed by the host gracefully shutting down fully. The CyberPower unit then kills power to the outlets until AC power has been restored for 5 minutes stably. Once AC is up, it restores power to the outlets. Since the motherboard is set to "power on" with AC restore, the host comes back up automatically and powers up the appropriate VMs. After all this time, it's finally working as planned!
     
    #36
    Daniel_GT, dawsonkm and Chuckleb like this.
  17. Daniel_GT

    Daniel_GT New Member

    Joined:
    Sep 25, 2014
    Messages:
    9
    Likes Received:
    0
    That seems to do the job, good spot, thanks.

    I've updated the z9pe-d8 BIOS to the latest version 5601 Motherboard - Z9PE-D8 WS - ASUS, additionally flashed firmware for the ASMB6 to 1.14 (not available through the official Asus page though some googling tracked it down ASUS ASMB6-iKVM drivers (page 5)
    ) - still no go without ZeroOne's workaround.

    Short of modifying the ipmi vib or Asus firmware I can't see any way further forward.
     
    #37
  18. ZeroOne

    ZeroOne Member

    Joined:
    Sep 13, 2013
    Messages:
    52
    Likes Received:
    6
    Latest update as of 3/7, the new BIOS on the Asus site, 1101 that was released yesterday for the P9D-E/4L along with the latest firmware, 1.09 for the ASMB-7 seem to be working. With IPMI re-enabled in ESXi 5.5, it seems to start up and shut down properly.
     
    #38
  19. Daniel_GT

    Daniel_GT New Member

    Joined:
    Sep 25, 2014
    Messages:
    9
    Likes Received:
    0
    @ZeroOne Would you share your BIOS config? I'm still having no luck with the ASMB6 and the Z9PE-D8, latest everything. Additionally, can you power on your server from the BMC?
     
    #39
  20. ZeroOne

    ZeroOne Member

    Joined:
    Sep 13, 2013
    Messages:
    52
    Likes Received:
    6
    Small update... after the server running for weeks, I randomly checked the performance chart for the host and found ~35% CPU usage with VMs doing nothing. It wasn't locked to one core in particular but distributed across all 4/8 (core/threads) to equal that ~35%. It appears there is another IMPI issue. Have not yet moved to ESXi 6 yet. Disabling IPMI in ESXi again for now. It did shut down properly during this condition, but I'd rather not waste that much power/heat for no real reason, considering I log in directly to the motherboard vs. relying on VMware to report on hardware health.

    The only steps I took when setting up the bios were to disable all unused ports, make sure VT/d was enabled, disable option rom for nics & cards, set boot order, fan speeds, etc. Is there any specific page or settings you were unsure of? I can check them.
     
    #40
Similar Threads: Can't shut
Forum Title Date
VMware, VirtualBox, Citrix ESXi 6.7 can't change settings, don't survive reboot (passthrough issues, too!) May 1, 2018
VMware, VirtualBox, Citrix Can't get Ubuntu to install in ESXI 6 Sep 11, 2016
VMware, VirtualBox, Citrix Can't get passthrough to work on esxi 6.0 with connectx-3 May 31, 2016
VMware, VirtualBox, Citrix ESXi 5.5 - Can't connect to vSphere Data Protection Appliance Apr 8, 2015
VMware, VirtualBox, Citrix ESXI Sys Reset on Win10 Guest Shutdown Nov 4, 2017

Share This Page