Backup Server Powering On By Itself?

eptesicus

Member
Jun 25, 2017
95
13
8
33
NAS02 Server specs:

OS: Windows Server 2016
Chassis: Supermicro SC846E16-R1200B
PSU: Redundant 920W PWS-920P-1R 80PLUS Platinum
Mobo: ASRock E3C224 ATX
CPU: Intel E3-1241v3 3.5GHz
RAM: 32 GB Kingston 4x8 GB DDR3 ECC Unbuffered KVR16E11/8
NIC: Mellanox ConnectX-2 (Directly connected to my primary NAS, NAS01)
RAID: IBM M5015 w/ BBU
SSD: 120 GB Samsung 840 Evo - OS
HDD: 14x 4 TB HGST NAS (Raid 6) - Primary backup pool


This server is my backup server of a similarly spec'd NAS (NAS01) that houses all of my... stuff. NAS02 stays powered off throughout the day, but at 01:00 every night, a scheduled task on NAS01 runs a script to power on NAS02 via IPMI (if it's not already on), waits for NAS02 to come online, and then executes a GoodSync job to copy any new or modified data to NAS02. After the backup job completes, the script on NAS01 will send a shutdown command to NAS02, but only if it was powered off already before the script executed. I'm not the best at scripting, but it works well...

HOWEVER... Every night, after the backup job finishes, and NAS02 has shutdown, anywhere between 20 minutes and a couple hours later, NAS02 will power back on by itself. I've checked the Windows event viewer, and it shows nothing other than a standard power-on, as if someone powered on the server manually. I checked the logs in my motherboard's IPMI, and also see a normal power-on. The logs don't show if or what user powered on the server (below). Even when I power on the server in the IPMI console manually, using my admin account, it doesn't log and say that the admin sent the power-on command.

09/15/2017 02:47:22 Unknown OS Boot C: Boot Completed - Asserted

I'm at a loss. I have no idea why I'm getting these phantom boots on this server. I went ahead and set the 'after power failure' setting in my BIOS to 'Off', as it was 'last state' before. This didn't fix it either. I have two Cyberpower UPS' for three servers (my host running the center install), and looked in the center's and client's logs, but also don't see any indication that the power-on came from the UPS software.

What do I try next?

My scripts and last night's log is below.

7xNAS02-GSync-Script.bat (pardon the mess)
Code:
@echo off
REM Change CMD window color and size.
color 37
mode 180,50

REM ### Set variables
SET ipaddress1=11.11.11.20 REM ### Use the 10GbE NIC to ping 7xNAS02
SET ipaddress2=10.0.3.1
SET LOGFILE=C:\!SCRIPTS\7xNAS02-GSync-Log.txt
set DATE=%DATE:~4,10%
set dateStr=%DATE:~6,4%%DATE:~3,2%%DATE:~0,2%

:START
    REM    ### Checks if 7xNAS02 is online. If not, power it on.
    ECHO ################################################################################################################################ >> %LOGFILE%
    ECHO %DATE% %time%    EXECUTING GSYNC SCRIPT >> %LOGFILE%
    ECHO ################################################################################################################################ >> %LOGFILE%
    ECHO.
    ECHO This is an automated script to backup the storage array on 7xNAS01 onto 7xNAS02.
    ECHO If 7xNAS02 isn't powered on, the script will boot 7xNAS02, wait for it to come online,
    ECHO and then execute the "STORAGE" backup job in GoodSync.
    ECHO.
    ECHO %DATE% %time%    Checking power status of 7xNAS02... >> %LOGFILE%
    ECHO Checking power status of 7xNAS02...
    ECHO.
    ping -n 1 %ipaddress1% | findstr TTL && goto ALREADY_ON
    ping -n 1 %ipaddress1% | findstr TTL || goto POWER_ON


:ALREADY_ON
    REM Sets variable to dictate that the server was already on before executing the script.
    ECHO.
    ECHO %DATE% %time%    7xNAS02 is already powered on and available. >> %LOGFILE%
    ECHO 7xNAS02 is already powered on and available.
    ECHO.
    SET previously_off=1
    goto EXECUTE_BACKUP


:POWER_ON
    REM ### Powers on 7xNAS02
    ECHO %DATE% %time%    7xNAS02 is offline. Powering on now. >> %LOGFILE%
    ECHO 7xNAS02 is offline. Powering on now.
    ECHO.
    SET previously_off=0
    PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& 'C:\!SCRIPTS\7xNAS02-Power-On.ps1'" >> %LOGFILE%
    ECHO %DATE% %time%    Pinging 7xNAS02 until it's available... >> %LOGFILE%
    ECHO Pinging 7xNAS02 until it's available...
    ECHO.
    goto CHECK_IF_READY


:CHECK_IF_READY
    REM ### Continually pings 7xNAS02 until it is online.

    ping -n 1 %ipaddress1% | findstr TTL || goto CHECK_IF_READY
    ping -n 1 %ipaddress1% | findstr TTL && goto EXECUTE_BACKUP


:EXECUTE_BACKUP
    REM ### Executes GoodSync backup job, "STORAGE."
    ECHO %DATE% %time%    Executing "STORAGE" backup job. >> %LOGFILE%
    ECHO Executing "STORAGE" backup job.
    ECHO.
    c:\"Program Files"\"Siber Systems"\Goodsync\gsync sync "STORAGE" >> %LOGFILE%
    if %previously_off% == 0 (
        ECHO.
        ECHO %DATE% %time%    Sync job done. Shutting down 7xNAS02... >> %LOGFILE%
        ECHO Sync job done. Shutting down 7xNAS02...
        shutdown /m \\7xNAS02 /s /f /d p:0:0 /c "Executed by 7xNAS02-GSync-Script on 7xNAS01."
        goto SHUTDOWNCHECK
       
    )
    if %previously_off% == 1 (
        ECHO.
        ECHO %DATE% %time%    Sync job done. 7xNAS02 will stay powered on. >> %LOGFILE%
        ECHO Sync job done. 7xNAS02 will stay powered on.
        ECHO.
    )
    goto END

   
:SHUTDOWNCHECK
    ping -n 5 %ipaddress1% >nul
    if ERRORLEVEL 1 (
        REM ### 7xNAS02 has powered off.
        ECHO.
        ECHO %DATE% %time%    7xNAS02 has powered off. >> %LOGFILE%
        ECHO 7xNAS02 has powered off.
        goto END
    )
    REM ### 7xNAS02 is still on. Looping to check if the server has shut down.
    ECHO ...
    goto SHUTDOWNCHECK  
   

:END
    REM pause
    ECHO %DATE% %time%    END OF SCRIPT >> %LOGFILE%
    exit
7xNAS02-Power-On.ps1
Code:
$username = "username"
$password = "password"
$secstr = New-Object -TypeName System.Security.SecureString
$password.ToCharArray() | ForEach-Object {$secstr.AppendChar($_)}
$cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $username, $secstr
$Ip = "10.0.3.99"
Get-PcsvDevice -TargetAddress $Ip -ManagementProtocol IPMI -Credential $Cred | Start-PcsvDevice
Log output from script:
Code:
################################################################
09/15/2017  1:00:00.86    EXECUTING GSYNC SCRIPT
################################################################
09/15/2017  1:00:00.87    Checking power status of 7xNAS02...
09/15/2017  1:00:08.82    7xNAS02 is offline. Powering on now.
09/15/2017  1:00:12.20    Pinging 7xNAS02 until it's available...
09/15/2017  1:02:20.92    Executing "STORAGE" backup job.
*REDACTED*
09/15/2017  2:24:13.30    Sync job done. Shutting down 7xNAS02...
09/15/2017  2:25:33.82    7xNAS02 has powered off.
09/15/2017  2:25:33.82    END OF SCRIPT
EDIT: After posting this, I found that my BIOS has a Boot from Onboard LAN feature that was disabled, so it wasn't that. However, I did see that 'Allow this device to wake the computer' was checked in my NIC's power management settings. I went ahead and disabled that.
 
Last edited:

Blinky 42

Active Member
Aug 6, 2015
610
225
43
45
PA, USA
In your bios is there anything for a scheduled wakeup? I have seen pages with every day at X turn on (so sw installs can be done etc) there might be something like that hiding in the menus.
 

eptesicus

Member
Jun 25, 2017
95
13
8
33
In your bios is there anything for a scheduled wakeup? I have seen pages with every day at X turn on (so sw installs can be done etc) there might be something like that hiding in the menus.
RTC Alarm Power On was enabled. I went ahead and disabled that. We'll see if that and the NIC's power settings change anything tonight.
 

Tom5051

Active Member
Jan 18, 2017
290
50
28
43
RTC Alarm Power On was enabled. I went ahead and disabled that. We'll see if that and the NIC's power settings change anything tonight.
RTC = Real Time Clock. Looks like you had an alarm to wake the machine. Theeeeeeere's ya problem
 

Tom5051

Active Member
Jan 18, 2017
290
50
28
43
RTC Alarm Power On was enabled. I went ahead and disabled that. We'll see if that and the NIC's power settings change anything tonight.
RTC = Real Time Clock. Looks like you had an alarm to wake the machine. Theeeeeeere's ya problem
 

eptesicus

Member
Jun 25, 2017
95
13
8
33
Sooo.... I thought that this had resolved, but it's not. I'm out of ideas, but the last few nights, the server boots back up again at around 2:45AM. Any thoughts here? I'm tempted to reset the bios and IPMI again, but don't know if this will fix it.
 

Rand__

Well-Known Member
Mar 6, 2014
5,522
1,191
113
If you're desperate you always can use a hardware timer;)

Else it sure sounds like the issue is not on the box - have you checked other systems to ensure they are not sending wake up packets?
 

eptesicus

Member
Jun 25, 2017
95
13
8
33
It woke up by itself again last night... Everything on my network has that is capable of WOL has it disabled (on every NIC and in the BIOS), and I can't think of anything that would be sending those packets. I went ahead and did a reset of my BIOS settings again, just in case. I'm tempted to contact ASRock to see if they'd know of anything that would be causing this.
 

markarr

Active Member
Oct 31, 2013
417
115
43
I would try unplugging any network connection, to rule out anything sending WOL or you ipmi script running unexpectedly, so you can isolate it to the box itself.
 

eptesicus

Member
Jun 25, 2017
95
13
8
33
I would try unplugging any network connection, to rule out anything sending WOL or you ipmi script running unexpectedly, so you can isolate it to the box itself.
I had thought about this... Now, the script I'm running doesn't run outside of the scheduled time that I can see. Nothing in the event logs or the logs created by my script indicate that it's running again. But now I'm thinking, is there another way or better way to power on the server remotely without using WOL?
 

Rand__

Well-Known Member
Mar 6, 2014
5,522
1,191
113
As i said, get a hardware timer;)
Set to always powered on in bios, shutdown after backup in Windows and off you go (pun intended;))
 

eptesicus

Member
Jun 25, 2017
95
13
8
33
As i said, get a hardware timer;)
Set to always powered on in bios, shutdown after backup in Windows and off you go (pun intended;))
Haha. That sounds like a band-aid to the problem, but I appreciate the input. I'd rather try to find the source of the problem and fix it the right way.
 

Rand__

Well-Known Member
Mar 6, 2014
5,522
1,191
113
Well if you don't want to use WOL thats probably the only other sensible trigger.
But I agree its better to find the culprit;)

Try @markarr's idea with disconnected network to identify internal or external trigger
 

Blinky 42

Active Member
Aug 6, 2015
610
225
43
45
PA, USA
For kicks (or just out of desperation) try and replace the coin cell battery on the motherboard. I fit is near dead I could sorta see the mobo acting odd when the battery can't keep the RTC going.
 

eptesicus

Member
Jun 25, 2017
95
13
8
33
I went ahead and disconnected the 2x 1GbE ports and the 10 GbE port that's connected to 7xNAS01. I left the BMC/IPMI port connected for the night. We'll see what happens in the morning.
 

eptesicus

Member
Jun 25, 2017
95
13
8
33
With IPMI connected, I woke up this morning to the server powered on again. However, I'm an idiot and didn't think this through.... I forgot to disable my script to power on via IPMI... I'll have to shut it down and disable the script. Then the following night, I'll unplug the IPMI port.
 

Blinky 42

Active Member
Aug 6, 2015
610
225
43
45
PA, USA
When you unplug the ipmi port make sure it isn't set to fail over to the main nic if they are on the same network
 

eptesicus

Member
Jun 25, 2017
95
13
8
33
Well...

No NICs plugged in = no boot
All NICs, minus IPMI NIC = no boot
All NICs, w/ boot script disabled = no boot

I'm guessing that it's my powershell IPMI boot script. I'm tempted to move the scheduled task so that the script will run at 2:30. I'm curious if the server will boot back up again, a few hours later.