Can't shut down ESXi 5.5 host fully

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

ZeroOne

Member
Sep 13, 2013
52
6
8
Flashed the ASMB7-iKVM back to 1.05. Seems the CPU issue is fixed. The system still cannot be shut down with this installed though. Same hang on shutdown. Almost not worth it, to leave it installed, once the system is up and configured.


Update: 1.05 did not fix the issue. Removing the card entirely until a new update is available to test. Not sure if it's a system board bios, or the add on card firmware.
 
Last edited:

jingjing

Banned
Nov 23, 2013
22
0
0
sounds like the drivers for the motherboard are incomplete as far as power management, does it show in vcenter your power load under advanced performance ?
 

ZeroOne

Member
Sep 13, 2013
52
6
8
sounds like the drivers for the motherboard are incomplete as far as power management, does it show in vcenter your power load under advanced performance ?
The graph under the Performance tab, using the drop-down to select "power?" If so, not currently. It did before I removed the add-on card, though it showed like 300 watts, if I'm remembering correctly. Wall power was closer to 110w at that time, measured with an inline meter. After removing the add-on card, it now sits at 0w.
 

33_viper_33

Member
Aug 3, 2013
204
3
18
I've been eying this board since its release. Any luck with the shutdown issue? I see firmware version 1.08.2 is available forthe ASMB7-iKVM.
 

Borsaid

New Member
Aug 8, 2014
2
0
1
41
I've run into the same problem as ZeroOne, but with the ASUS P9D-I board. The ASMB7-IKVM is onboard as well. Although I have not been able to replicate the CPU spike issue on the ASMB7-IKVM Firmware v 1.08, I definitely can not shutdown ESXi cleanly.

I was hoping someone has had progress on this issue since February 2014.
 

Borsaid

New Member
Aug 8, 2014
2
0
1
41
Scratch that. Walked into the office this morning and AM having that exact same CPU spike issue. Downgraded the IKVM firmware this morning and rebooted.
 

mbau

New Member
Aug 27, 2014
2
0
1
52
Same problem here with ESXi 5.5, an ASUS Z9PA-D8 with a ASMB6 iKVM Controller.

ESXi not power off the host.
 

Daniel_GT

New Member
Sep 25, 2014
10
0
1
53
I'd like to chip in on this one. I'm using the ASMB6 with the 1.10 firmware, on a Z9PE-D8 WS board and ESXi 5.5 update 2.

Can confirm that, out of the box so to speak, the host will not perform a graceful shutdown. It seems to send the signal which starts the process and stalls right at the end, spinning up the fans. The host can be turned off at this point by sending a "Power Off - Immediately" command from the web client though this is patchy. I also rolled back at least two driver versions which didn't help.

I contacted ASUS support who referred me to their OS compatibility, which doesn't include ESXi - damn shame as most if not all of the other features work well. Considering that the last firmware version is a couple of years old I don't expect they'll be adding it anytime soon though I've found having a meaningful conversation with them quite difficult.

I'm wondering if some fine tuning of board BIOS settings for the interface AND power management (ACPI is it?) might lead to a solution. I've also noted that within vCenter Server an IPMI setting can be set (user, password, IP, MAC) though fails to communicate properly. VMWare KB suggests it's the protocol version.

I'd like to think that a solution can be found, feels to me that a fix should be possible? I'll update as I investigate further.
 

mbau

New Member
Aug 27, 2014
2
0
1
52
I checked with VMWare support the shutdown problem and they confirm that the issue is the ASMB6.

I also contacted the ASUS support, but there i not received a feedback till today. The funny thing is, is a server system TS700-X7 from ASUS on the compatibility guide from VMWare which also contains an ASMB6 and an ASUS Z9PA-D8 mainboard like i use.

I wonder if Asus uses there a special firmware which is not public available?
 

Daniel_GT

New Member
Sep 25, 2014
10
0
1
53
Hello,

there is a new beta BIOS from ASUS with version number 0803 which fixes the shutdown problem.
Thanks Ulrich, do you have a link you can share? Can't see anything on the official downloads page for either my mobo (see above) or the ASMB6?
 

ZeroOne

Member
Sep 13, 2013
52
6
8
Actually made some headway on this and can shut down now. Turns out, there's a flaw in ESXi. Specifically, a driver for IPMI built in that locks one core of the CPU up. Stumbled on this, of all places, on an Amazon review of another product. For what it's worth, I'm also running the latest motherboard bios from 4/21/2014 and ASMB7 firmware 1.08.2 from 1/20/2014.

Quoting the Amazon post:
VMWARE ESXi STABILITY NOTE: ESXi has an issue with many motherboards that have IPMI sensors on board where a runaway process will consume most of the CPU timeslices even without any guest OS running. A workaround for this problem is to go into Configuration -> Software -> Advanced Settings -> VMkernel -> VMkernel.Boot.ipmiEnabled and uncheck the box. Unfortunately you lose the ability to monitor temperatures, fan speeds, etc. with VMware (vSphere Client among others) software. Fortunately you can go into the web-interface for the IPMI and see the same information.
 

ZeroOne

Member
Sep 13, 2013
52
6
8
Just to reiterate, following the steps to disable the VMkernel.Boot.ipmiEnabled value does allow me to shut the host completely down as expected.

This, combined with the CyberPower appliance has things working perfectly. 60 minutes after the power goes out, the VMs shut down in succession followed by the host gracefully shutting down fully. The CyberPower unit then kills power to the outlets until AC power has been restored for 5 minutes stably. Once AC is up, it restores power to the outlets. Since the motherboard is set to "power on" with AC restore, the host comes back up automatically and powers up the appropriate VMs. After all this time, it's finally working as planned!
 

Daniel_GT

New Member
Sep 25, 2014
10
0
1
53
Just to reiterate, following the steps to disable the VMkernel.Boot.ipmiEnabled value does allow me to shut the host completely down as expected.

This, combined with the CyberPower appliance has things working perfectly. 60 minutes after the power goes out, the VMs shut down in succession followed by the host gracefully shutting down fully. The CyberPower unit then kills power to the outlets until AC power has been restored for 5 minutes stably. Once AC is up, it restores power to the outlets. Since the motherboard is set to "power on" with AC restore, the host comes back up automatically and powers up the appropriate VMs. After all this time, it's finally working as planned!
That seems to do the job, good spot, thanks.

Hello Daniel,

there is a new BIOS for the P9D-I with the ASMB7: http://dlcdnet.asus.com/pub/ASUS/mb/socket1150/P9D-I/BIOS/P9D-I-ASUS-0803.zip

Thought there would also be BIOS updates for all mobos with the ASMB6...
I've updated the z9pe-d8 BIOS to the latest version 5601 Motherboard - Z9PE-D8 WS - ASUS, additionally flashed firmware for the ASMB6 to 1.14 (not available through the official Asus page though some googling tracked it down ASUS ASMB6-iKVM drivers (page 5)
) - still no go without ZeroOne's workaround.

Short of modifying the ipmi vib or Asus firmware I can't see any way further forward.
 

ZeroOne

Member
Sep 13, 2013
52
6
8
Latest update as of 3/7, the new BIOS on the Asus site, 1101 that was released yesterday for the P9D-E/4L along with the latest firmware, 1.09 for the ASMB-7 seem to be working. With IPMI re-enabled in ESXi 5.5, it seems to start up and shut down properly.
 

Daniel_GT

New Member
Sep 25, 2014
10
0
1
53
@ZeroOne Would you share your BIOS config? I'm still having no luck with the ASMB6 and the Z9PE-D8, latest everything. Additionally, can you power on your server from the BMC?
 

ZeroOne

Member
Sep 13, 2013
52
6
8
Small update... after the server running for weeks, I randomly checked the performance chart for the host and found ~35% CPU usage with VMs doing nothing. It wasn't locked to one core in particular but distributed across all 4/8 (core/threads) to equal that ~35%. It appears there is another IMPI issue. Have not yet moved to ESXi 6 yet. Disabling IPMI in ESXi again for now. It did shut down properly during this condition, but I'd rather not waste that much power/heat for no real reason, considering I log in directly to the motherboard vs. relying on VMware to report on hardware health.

@ZeroOne Would you share your BIOS config? I'm still having no luck with the ASMB6 and the Z9PE-D8, latest everything. Additionally, can you power on your server from the BMC?
The only steps I took when setting up the bios were to disable all unused ports, make sure VT/d was enabled, disable option rom for nics & cards, set boot order, fan speeds, etc. Is there any specific page or settings you were unsure of? I can check them.