Asrock BMC no longer accessible after SSL certificate install

tfboy · Apr 1, 2023

Since purchasing an Asrock rack X570D4U-2L2T, it's been fine and the BMC IPMI has been really useful.
Today, I tried installing a certificate so have proper SSL (certificate generated from my own private PKI CA), and I must have done something wrong or it has serious QA issues as I can no longer access the web interace.

I can reach the IP over SSH (I get the Smashlite Scorpio Console), but that's about it.

Unfortunately, I'm physically away so cannot reset anything physically.

So I have two question:

1. Is there a way to reset / fix / return to the default self-signed certificate using the SSH console or do I have to wait till I have physical access to the machine again?

2. Clearly, I've had an issue with the certificate used. It was with a .pem extension for both the cert and the key and in PEM format. Are there any funny restrictions?

casperghst42 · Apr 4, 2023

When you say "not longer accessible", is that fully no access, or that the browser complains that the BMC no longer have a valid certificate?

RolloZ170 · Apr 4, 2023

tfboy said:
1. Is there a way to reset / fix / return to the default self-signed certificate using the SSH console or do I have to wait till I have physical access to the machine again?

for BMC reset without PW you need physical access. would be strange if not.

tfboy · Apr 4, 2023

casperghst42 said:
When you say "not longer accessible", is that fully no access, or that the browser complains that the BMC no longer have a valid certificate?

I get an error, cannot connect. Not an invalid cert.

RolloZ170 said:
for BMC reset without PW you need physical access. would be strange if not.

I have the password and can login over SSH. Is there a way to reset the BMC over SSH? I've found how to reset the server but not the BMC itself.

marcoi · Apr 4, 2023

try this support page from asrock
ASRock Rack > Support

RolloZ170 · Apr 4, 2023

tfboy said:
I have the password and can login over SSH. Is there a way to reset the BMC over SSH? I've found how to reset the server but not the BMC itself.

can you run a linux cmd ?

https://www.asrockrack.com/support/faq.asp?id=62

tfboy · Apr 4, 2023

RolloZ170 said:
can you run a linux cmd ?

https://www.asrockrack.com/support/faq.asp?id=62

Yes, but I'm completely flummuxed how this works.
Their website tells you to install ipmitool which is straight foward, then jumps straight to that second command.
No information how you connect to it!
I've tried ipmitool -H <BMC IP address> -U admin and then entering the password, but I just get :

Code:

Authentication type NONE not supported
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session

RolloZ170 · Apr 4, 2023

tfboy said:
heir website tells you to install ipmitool which is straight foward, then jumps straight to that second command.
No information how you connect to it!

run on local machine don't need connection path(ip i.e.)

tfboy · Apr 4, 2023

RolloZ170 said:
run on local machine don't need connection path(ip i.e.)

Ah. OK.
Only issue, I'm running vSphere on it, so no bare metal OS access.
I guess this is serial? If I can have a passthrough serial or something... I'll give it a try.

RolloZ170 · Apr 4, 2023

tfboy said:
Only issue, I'm running vSphere on it, so no bare metal OS access.

have seen many using ipmitool binary for ESXi.
ther should be a solution for vSphere thought.

tfboy · Apr 4, 2023

Aha.
I've had some success!
I stumbled across this site ipmitool 1.8.11 vib for ESXi
And managed to get it installed.
I ran the command and it just returned a prompt. I assumed it worked because shortly after, the IP side went down (ping dropped) and after a few minutes, came back up. So the BMC was reset!

But still no https interface

My guess is the reset doesn't restore the SSL side to the factory-installed self-signed certificate.
I might just have to wait till I can access the box physically to reset.

I have emailed Asrock support so will see what they suggest.

It's probably worth digging a little more with the ipmi, maybe another command can help me out

tfboy · Apr 4, 2023

Final update:
The reset was successful in that the user account and password was reset to admin / admin. I can se this with the ipmitool - if I have the wrong credentials, I get a Error: Unable to establish IPMI v2 / RMCP+ session message.

However, when I use the correct credentials, I'm still stuck:

Code:

$ ipmitool -H 192.168.18.200 -U admin -I lanplus sol activate
Password:
Error activating SOL payload: Unknown (0x18)

If I can't get this resolved remotely (looking unlikely now!), then I hope I can do a deeper reset than what I've been able to do so far as that doesn't reset the SSL certificates...

RolloZ170 · Apr 4, 2023

"I was able to set the password using ipmitool locally with the command "ipmitool -I open user set password 2 [password]", cf.)"

twin_savage · Apr 5, 2023

When/If you do a system reset, ensure the BMC actually has a MAC address after the reboot. I had an Asrock BMC "forget" its MAC address after an update once and all kinds of weird problems ensued until I re-established it.

tfboy · Apr 30, 2023

ok, took me a while to sort this out.
A full reset, clearing of the BIOS didn't help.
In the end, I had to do a full reflash of the BIOS. This was the only solution to this particular problem which I find a little worrying: supplying a certificate that, for some reason, it doesn't like, completely crashes the web server side to the point that a full re-writing of the flash is required and not even a full BMC BIOS reset will work.
The flashing takes a while too...

scyto · Dec 15, 2024

Did you ever figure out a way to do this how you wanted?
I am hoping to find a way to use acme.sh to poke a certificate and key into my asrock bmc via ipmi, i found this for supermicro, but i am pretty sure this is not portable to asrock GitHub - gnarr/ipmi-cert: Supermicro IPMI certificate updater

agross · Dec 16, 2024

I have a script that uses curl and the ASRock BMC API. It is invoked as a deploy hook by certbot. It likely needs adjustment for acme.sh. I consider it a hack, but it works well with a X570D4U-2L2T board.

Bash:

#!/usr/bin/env bash

set -euo pipefail

bmc=https://172.16.0.253
bmc_user=some-admin
bmc_password=secret
curl_cookie_jar="$(mktemp)"
curl=(curl --insecure --silent --show-error --fail --cookie-jar "$curl_cookie_jar" --cookie "$curl_cookie_jar")

# RENEWED_DOMAINS is set by certbot after obtaining a new cert.
for domain in $RENEWED_DOMAINS; do
  case $domain in
    example.com)
      >&2 printf "Updating cert on BMC for domain %s\n" "$domain"

      if ! token="$("${curl[@]}" \
                         --data-raw "username=$bmc_user&password=$bmc_password" \
                         "$bmc/api/session" \
                      | jq --raw-output .CSRFToken)"; then
        >&2 printf 'Login failed\n'
        exit 1
      fi

      >&2 printf 'Logged in with CSRF token %s\n' "$token"

      "${curl[@]}" \
           --header "X-CSRFTOKEN: $token" \
           --header "Content-Type: application/json" \
           --data-raw '{"uid_command":1}' \
           "$bmc/api/actions/chassis-led"

      >&2 printf '\nUID LED blinks temporarily\n'

      "${curl[@]}" \
           --header "X-CSRFTOKEN: $token" \
           --form "new_certificate=@$RENEWED_LINEAGE/fullchain.pem" \
           --form "new_private_key=@$RENEWED_LINEAGE/privkey.pem" \
           "$bmc/api/settings/ssl/certificate"

      >&2 printf '\nDone\n'
    ;;
  esac
done

scyto · Dec 16, 2024

agross said:
I consider it a hack, but it works well with a X570D4U-2L2T board.

What you consider a hack i consider pragmatically elegant

thanks, that looks simple enough, i will be using the neilpang acme.sh container so i am sure i can figure it out, might have to wait for a few months as i will shrtly be out of action for 6 weeks). [edit, heck i can check certbot too]

thanks for sharing, really appreciate it

just as this might tickle you interest, i did find this after i posted, but a)super complext and b)super complex so i am glad to have what you shared

GitHub - sapcc/redfish-certrobot: A ACME DNS-01 robot using Redfish to maintain certificates on BMCs

A ACME DNS-01 robot using Redfish to maintain certificates on BMCs - sapcc/redfish-certrobot

github.com

(it uses the redfish API to generate a CSR, the passes that to LE using an acme client called lego, then uploads the resulting cert [and it does it for all the machines they have stored in some asset system, very clever, too clever for me)
this is cool, that you now educated me there is a /api endpoint in addition to the /redfish/v1/CertificateService endpoint too. The API endpoint looks better as you can upload the key, the redfish endpoint doesn't seem to have that

scyto · Dec 29, 2024

thanks @agross and i was suprrised that copilot got me from his script to this, which is working (run manually after running acme.sh docker container... i will find out in Feb if it worked as --renew_hook.... i need to convert the script from bash to sh (that was 4 things and co-pilot did all of that and it suggested the code to iterate through the dirs, and showed me how to derive the domain name and certname from the directory name when it gave an example (for reference i cant write a lick of code from scratch). Next up to make this into a docker stack that uses scret for the user and the name.

I leave this here as-is here as it does the main thing for a prototype - takes certs from an acme.sh output folder and uploads it to the BMC, as i think about it more, iterating though the dirs make no sense, better to place in output dir, capture a $PWD and use that to inform that paths, will refactor it later....

Bash:

#!/bin/sh

# THIS SCRIPT SHOULD BE PLACED IN THE /ACME.SH FOLDER
set -eu

bmc=https://192.168.1.118
bmc_user=some user
bmc_password=some admin
curl_cookie_jar="$(mktemp)"
curl="curl --insecure --silent --show-error --fail --cookie-jar $curl_cookie_jar --cookie $curl_cookie_jar"

#at the moment it iterates for the subdir and processes files in every folder with _ecc - either this should be changed to look for a specifc folder called by the --renew-hook command, or by a variable, bur for now this is what it is, or the script could be place in the _ecc folder and called by hand - i need to think about that

for dir in */; do
#  if [ -d "$dir" ]; then
if [ -d "$dir" ] && echo "$dir" | grep -q "ecc"; then
    echo "Directory: $dir"

    # Extract the base directory name without the trailing slash and "_ecc"
    base_dir=$(basename "$dir" _ecc)

    printf "Updating cert on BMC for domain %s\n" "$base_dir" >&2

    token=$($curl --data-raw "username=$bmc_user&password=$bmc_password" "$bmc/api/session" | jq --raw-output .CSRFToken)
    if [ -z "$token" ]; then
      printf 'Login failed\n' >&2
      exit 1
    fi

    printf 'Logged in with CSRF token %s\n' "$token" >&2

    $curl --header "X-CSRFTOKEN: $token" \
          --header "Content-Type: application/json" \
          --data-raw '{"uid_command":1}' \
          "$bmc/api/actions/chassis-led"

    printf '\nUID LED blinks temporarily\n' >&2

    $curl --header "X-CSRFTOKEN: $token" \
          --form "new_certificate=@./$dir/fullchain.cer" \
          --form "new_private_key=@./$dir/${base_dir}.key" \
          "$bmc/api/settings/ssl/certificate"

    printf '\nDone\n' >&2
  fi
done

Asrock BMC no longer accessible after SSL certificate install

New Member

Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

New Member

Well-Known Member

Active Member

New Member

New Member

New Member

New Member

New Member