Supermicro X10SDV-TP8F IPMI settings broken

[fckruegel]

Hi!

I recently got two Supermicro X10SDV-TP8F systems. One works just fine, the other one behaved oddly. I started to note that the IPMI http showed the last system ethernet MAC addresses as ff:ff:ff:ff:ff:ff, and others with random values, while BIOS shows the correct values. I set the system to remain off on power-up. The BIOS showed the correct setting, but the system would power on automatically when getting mains power.

Ok, then. I downloaded the latest BIOS and IPMI/Redfish firmware, updated both while not preserving any setting.

Things got worse then. IPMI does not show any MAC addresses in the intro screen. I can't add additionsal users since the user list does not show. I can't mount any ISO images since I can't enter the required data. I managed to set IP address through BIOS, and the only account that works is the ADMIN/ADMIN account.

How can I restore the system properly?

Thanks.

Frank.

 

[oneplane]

Looks like your IPMI software might not be able to communicate with the BIOS/UEFI software or the SPI Flash. I usually see this with mis-matched versions.

Are you running REDFISH_X10_388_20200221_unsigned.zip (3.88) in combination with BIOS_X10SDVT_20210507_2.3_STDsp.zip (2.3) ?

 

[fckruegel]

Yes.

19.04.2022 11:53 5.452.275 BIOS_X10SDVT_20210507_2.3_STDsp.zip
19.04.2022 11:53 27.375.931 REDFISH_X10_388_20200221_unsigned.zip

These are the files I uploaded.

Frank.

 

[oneplane]

What versions is the known-good board running?

 

[fckruegel]

Firmware Revision : 03.72
Firmware Build Time : 05/10/2018
BIOS Version: 2.0
BIOS Build Time: 06/13/2018
Redfish Version : 1.0.1

I don't know where to get these versions.

 

[oneplane]

Supermicro does make it pretty hard but we do have ways to get what we need:

REDFISH_X10_372.zip can be found at https://www.supermicro.com/Bios/softfiles/5696/REDFISH_X10_372.zip

The BIOS is a bit more problematic because SuperMicro has like 3 different version numbers per file. There is a file version, a firmware version and a release version and none of them truly correlate...

Luckily for us, there is enough verification in place to make it really hard to brick a board, and as long as IPMI/BMC works you can even flash a bricked board. Because it's a Xeon D board there aren't that many options so it's really hard to go wrong here.

One option before downgrading your BMC is to use this two-in-1 package:
https://www.supermicro.com/Bios/softfiles/10169/X10SDVT_BIOS_2_1_IPMI_3_86.zip

It contains a matching (slightly older) pair for both firmwares. All 8F Xeon D boards use the SDVT firmware.
If those don't do it, you can try any older version:

X10SDVT0_212
X10SDVT7_A31

but a restriction on most (that you really shouldn't override) is downgrade protection on incompatible core versions. So if the flasher says you really shouldn't downgrade because the core version is too old, don't force it. The worst thing that can happen is that it loses more MAC address and UUID data, so it's not like things will go bad (not beyond a simple CMOS reset that will fix it) but why break it and then waste time fixing it If the core version isn't too old you can go bananas and downgrade as low as you want.

One aspect on firmware changes that is more important than whatever the IPMI/BMC is saying: make sure your OS agrees with the stickers on the mainboard. As long as your network interfaces still report the correct MAC addresses for example, the system firmware is fine and it's just the BMC firmware that is having a problem.

 

[i386]

Hi!

I recently got two Supermicro X10SDV-TP8F systems. One works just fine, the other one behaved oddly. I started to note that the IPMI http showed the last system ethernet MAC addresses as ff:ff:ff:ff:ff:ff, and others with random values, while BIOS shows the correct values. I set the system to remain off on power-up. The BIOS showed the correct setting, but the system would power on automatically when getting mains power.

Ok, then. I downloaded the latest BIOS and IPMI/Redfish firmware, updated both while not preserving any setting.

Things got worse then. IPMI does not show any MAC addresses in the intro screen. I can't add additionsal users since the user list does not show. I can't mount any ISO images since I can't enter the required data. I managed to set IP address through BIOS, and the only account that works is the ADMIN/ADMIN account.

How can I restore the system properly?

Thanks.

Frank.

did you reset the bmc after updating the bios or ipmi?

 

[fckruegel]

Congratiulations! You made my day!

I managed to find the exact BIOS version X10SDVT8.613.zip that was on my system. 8 means 2018, 6 is the month, and 23 is the day. Matches exactly the BIOS build date. Flashed this first from a Freedos USB Stick, Poweroff, Poweron, then used the same USB stick for the redfish_X10_372 without retaining settings ( -r n), reboot, and everything was back to normal.

Thank you very much!

Frank.

 

[oneplane]

Since you now have a successful recovery method you could try to update one of them slightly as there are some redfish security patches in there that are useful to have (as well as some USB3 and boot device fixes). One of those pre-packaged combo versions could be the right fit.

I remember one of my X10's having the same issue which was mostly due to SuperMicro upgrading the core firmware package from their IBV which is incompatible with all previous versions. First re-patching to a combo package that is new enough fixes that usually (they usually have one special version that is able to translate between the old nvdata and new nvdata).

On the other hand; if it works it works. Depending on your security environment, additional patching might not be a hard requirement.