Help, Gigabyte MD71-HB0 motherboard hangs after 1 day

[spacesaver]

Ugh, I've been struggling with these symptoms for days:
* after ~1 day of leaving the server on and idling, the SSH connection drops and network is completely unresponsive
* IPMI still stays on. The web based Remote KVM says "No Signal"
* CPU fans stay on
* sensors seem normal (attached). VRMs for 2nd processor do seem a little low @1.58V
* (smoking gun?) IPMI event log show power button pressed followed by power button released 5s later. The last occurrence for sure was unexpected and happened very close to when I noticed it get stuck. Don't know about the previous ones.

I've tried the following independently, but not neccesarilly all combinations. Each step below includes all previous steps
* upgrade to latest R27 BIOS. Load BIOS defaults. Reset IPMI settings.
* use the Marvel 1G ethernet port instead of Intel 10G
* change power supplies
* use 10G ethernet
* disconnect the power & reset switches (because of the unexpected power off event) - these can't randomly trigger unless pressed right? It still became unresponsive after ~ 1 day, but this time no power off/on event in IPMI log.


CPU: 2x Intel 8259CL
RAM: 12x 4 GiB DDR4 @2400MHz
Fans: 4000 RPM DELTA QUR0912VH + 3D printed duct
OS: Ubuntu 22 & kernel 6.8.138

 

[spacesaver]

Some progress. I found plugging in a PCIe display adapter (because I couldn't find any option to disable ASPEED VGA in BIOS) and stopping the Ubuntu desktop, has allowed it to run for > 1 day. Got this idea from seeing "No Signal" in the virtual KVM, which shouldn't happen. Any explanations?

 

[RolloZ170]

my EPYCD8-2T BMCs VGA stopped work years ago, this year also the complete AST2500 is dead.

 

[spacesaver]

Good to know. This motherboard does use AST2500. But so do many LGA3647 Supermicro ones and I've never had a problem. So don't have any evidence of a systematic problem yet. I'm going to see if I can prevent the hang by preventing the Linux ast driver from loading (I can't disable from BIOS).

 

[spacesaver]

Ugh, it's not easy to disable Linux writing to VGA. Even if you don't load the ASPEED or even VGA frame buffer driver, it still writes to the VGA text mode, I think, because I still see a console in the remote KVM. I ended up using this kernel command line

modprobe.blacklist=ast,vga16fb console=ttyS0,115200 systemd.mask=getty@tty1.service

Finally, there's no output to the remote KVM after booting. But it still hangs after 1 day. Why?
Only plugging in a PCIe graphics board prevents the hang.