H12SSL-i IPMI Status: Not working

[thulle]

I joined the plenty of people ordering a H12SSL-i + 7302p from ebay to replace my old dual X5650. Also found 4 sticks of MTA36ASF8G72PZ-3G2B2UI.
Everything arrived a few days ago and I mounted everything and connected a VGA monitor and a Ubuntu livecd on a Samsung SSD via USB->SATA adapter.

It ran memtest86+ for 12 hours, 2½passes, no issues. After that I connected 2x 250GB SATA SSD:s and installed ubuntu on a ZFS mirror, using rEFInd + ZFSBootMenu to boot it.
It booted fine, so I ran prime95 for 12 hours to check the CPU. No issues. So I moved the system closer to my desk and connected a network cable to the IPMI port and a cable to the first interface to start to migrate things. No monitor, but IPMI got an IP via DHCP, and after figuring out that my firefox security settings gave "Session timed out"-errors I logged in with chromium instead. The dashboard didn't give all sensor data, so I updated the BMC with BMC_H12AST2500-ROT-2201MS-GPIO01_20220524_01.01.06_STDsp.zip - but skipped BIOS update since it was recommended against by supermicro, unless I had some issues.

After that the sensor data was displayed just fine. I configured an IP-address in linux via IPMI, and then used SSH to connect and continue the configuration of libvirt and so on. After that I shut the power off and moved the Optane 900p, 1080TI GPU and two m1015-equivalents from the old server to the new and connected the monitor again, since a VM on the old server was acting DHCP server.
No image.
Of course, BIOS must be set to external GPU first, so I grab an HDMI-cable and connect the monitor to the 1080TI instead.
Much slower boot than before, but I get an image, BIOS flashes by and rEFInd starts, and goes:

---8<---
Starting vmlinuz.old-2.1.0_1
Using load options 'initrd=EFI\zbm\initramfs-2.1.0_1.img
EFI stub: Loaded initrd from command line option
_
---8<---

And then the keyboard goes black and it freezes. So I reboot, check the VGA output priority setting, and it shows as internal first. Odd. Maybe it's showing a default without being written to CMOS? So I switch it to external first, save & reboot, then enter bios again and set it to internal first. Same thing.
So I enter BIOS again, and set a static IP & netmask, no gateway, to IPMI to check out what it's saying, and reboot again. Time passes, and the IP never starts to reply to ping requests. Maybe the gateway was a required setting, even if it's not being used? Back into bios, and now it says:

BMC Firmware Revision: Unknown
IPMI STATUS: Not Working

Now I'm thinking maybe BIOS has to be updated in tandem with the BMC, but I didn't read anything about it, and the BMC didn't say anything about when i updated it via IPMI.

I cut power for all hardware changes. First removing all addon-cards, so I should be back in a working configuration, but no image via VGA. I remove all but 1 memory stick, same thing. Try another stick in another slot, same thing. So I add the 1080TI back again, and I get an image over HDMI.
I remove it again, clear CMOS, and switch to another battery while I'm at it. No changes.
I try booting the Ubuntu livecd over USB again. It loads GRUB, but hangs as soon as it starts to load the kernel, powering off the keyboard just as before.

Maybe this warrants a BIOS update, so I download latest BIOS_H12SSL-1B95_20220414_2.4_STDsp.zip and copy the contents to the EFI-partition of one of the linux drives I installed on earlier, and plug this in the USB->SATA adapter. Reading the help output for SUM.efi I realize it wants the IPMI password to flash a new BIOS, and thus as expected, it fails:
---8<---
FS0:\EFI\> flash.nsh BIOS_H12SSL-1B95_20220414_2.4_STDsp.bin ADMIN {{PASSWORD}}
[copyright header]

<<<ERROR>>>>
ExitCode = 101
Description = Driver input/output control failed
Program Error Code = 17.2
Error message: Send IPMI failed
---8<---

So, BMC seems broken, and I can't boot anything past UEFI-stuff anymore. Any Ideas?
No cratered IC as in the thread about "BMC Initiating", should I add pictures of the board?

I tried 2 other GPUs and another PSU too, just in case, no difference.

 

[RolloZ170]

Maybe this warrants a BIOS update, so I download latest BIOS_H12SSL-1B95_20220414_2.4_STDsp.zip and copy the contents to the EFI-partition of one of the linux drives I installed on earlier, and plug this in the USB->SATA adapter. Reading the help output for SUM.efi I realize it wants the IPMI password to flash a new BIOS, and thus as expected, it fails:

the EFI just uploads the BIOS to the BMC flashdisk and restarts system to let the BMC make the BIOS flash.
you could have done the BIOS update in IPMI as it was working before.

 

[thulle]

the EFI just uploads the BIOS to the BMC flashdisk and restarts system to let the BMC make the BIOS flash.
you could have done the BIOS update in IPMI as it was working before.

Of course I could have, but it was recommended against.

 

[RolloZ170]

...and FW updates pls only with the BMC VGA connection. not GPU installed. many issues reported because the flasher can't find the interface to BMC.

 

[RolloZ170]

Of course I could have, but it was recommended against.

have you read the notes form BMC update ? maybe a min. rev BIOS is required to work.

 

[RolloZ170]

maybe you have restarted the system to early. the tools say "operation success" but the action begins in background and a reset bricks the system.

 

[RolloZ170]

is the BMC Heartbeat LED blinking ?

 

[thulle]

As stated, the BMC updated fine, I used it afterwards, and addoncards, including GPU, were added after the BMC update.
BMC heartbeat LED is blinking for like 8 seconds or so, then it turns on as the system boots.

edit: Nothing in the BMC release notes about BIOS requirements, just a lists of fixes. It specifically says:
Dependencies: N/A

 

[RolloZ170]

BMC heartbeat LED is blinking for like 8 seconds or so, then it turns on as the system boots

must be blinking about 30 seconds after AC plug in. if it stops blinking indicates some action or error.
are you sure you located the BMC heartbeat LED right ?

 

[thulle]

Are you sure it shouldn't continue blinking indefinitely? Manual doesn't say anthing about solid light at all:
LEDM1 BMC heartbeat LED Green: Blinking (BMC normal), Green: Fast blinking (BMC initializing)

 

[RolloZ170]

 

[i386]

...and FW updates pls only with the BMC VGA connection. not GPU installed.

I did at least two bios updates on my h12ssl with an external gpu, no problems here. (I didn't buy a used mainboard but a new one from a retailer though...)

so I updated the BMC with BMC_H12AST2500-ROT-2201MS-GPIO01_20220524_01.01.06_STDsp.zip

I had to much coffee; did you use the zip file? Or did you unpack it?

 

[thulle]

Ah, further down. But yes, it's not blinking as it should. Aligns with BIOS saying it's not working.

 

[thulle]

I had to much coffee; did you use the zip file? Or did you unpack it?

Haha, unpacked, of course. I'd be impressed if it ran with a .zip as firmware.

 

[RolloZ170]

I did at least two bios updates on my h12ssl with an external gpu, no problems here. (I didn't buy a used mainboard but a new one from a retailer though...)

should work but....its just a general rule.

 

[thulle]

I didn't buy a used mainboard but a new one from a retailer though...

The H12-boards are listed as new BTW, CPUs as used. No way to verify it, but no signs of use, no wear around the holes for screws or similar.

 

[i386]

I would try to update the bios, if necessary with the super.rom approach.
If the system can boot to any os after the update: check if the bmc shows up as a device in the os
if the bmc is found: (re)flash/update the bmc firmware locally (not via ipmi and ip address)
if the bmc is not found: check the bios settings for the bmc and if it's doing something funny

 

[thulle]

I would try to update the bios, if necessary with the super.rom approach.
If the system can boot to any os after the update: check if the bmc shows up as a device in the os
if the bmc is found: (re)flash/update the bmc firmware locally (not via ipmi and ip address)
if the bmc is not found: check the bios settings for the bmc and if it's doing something funny

SUPER.ROM doesn't seem to be a thing on the H12 boards any longer? Just tried, it can find the USB stick, but it doesn't pick up a new BIOS. So, no success in booting an OS.

 

[RolloZ170]

SUPER.ROM doesn't seem to be a thing on the H12 boards any longer?

you mean you haven't ever visited the manual ?

Notes:
• The BIOS recovery (SUPER.ROM) will not work if BIOS has a multi-block crash.
• If you have purchased an OOB license, you can do a BIOS flash again through BMC WebGUI or SUM. Please refer to these SUPERMICRO software-related videos:
• Activate OOB license:

• SUPERMICO BIOS Update:

• If the BIOS flash recovery fails, contact our RMA Department to have the BIOS chip reprogrammed. This will require shipping the board to Supermicro for repair.
• Submit your RMA request at RMA Services | Supermicro
• Please make sure to follow all instructions when returning the motherboard.

 

[thulle]

you mean you haven't ever visited the manual ?

Who does that any more? Nah, I'm just saying it didn't pick it up at all. But I can see the drive in BIOS. Currently reformating the drive to FAT16 instead to see if that makes any difference. Since they left instructions on how to fit 16 DIMMS on a 8 DIMM board I thought Super.ROM might be some old remnant too.

• If you have purchased an OOB license, you can do a BIOS flash again through BMC WebGUI or SUM

How do I connect remotely if it doesn't reply to ARP requests?