Try:
Legacy USB Support - Disabled
Port 60/64 Emulation - Disabled
USB KB/MS Wake - Enabled
The last one is interesing.
Booting with the options as follows makes no difference.
Also, try hotplugging. Start with NO USB Devices plugged on, get a Terminal with dmesg -w to monitor changes then plug them.
There is no output from `dmesg -w` when unplugging and replugging a keyboard or flash drive into any of the rear USB ports, or the internal onboard port.
Since you mention it works in UEFI and you mention flashing via DOS, the USB hardware works, so it's probably something post-firmware that breaks it. Considering there are _OSC calls to ACPI and the firmware is likely an older UEFI/CSM implementation, have you tried booting an OS with ACPI turned off completely? And since Linux seems to know this hardware needs handoff quirks (and takes a huge amount of CPU time to do so), it is possible that you need to unload and re-load the USB kernel modules once the _OSC calls are done.
I tried booting with `noacpi acpi=off`. Made no difference unfortunately.
Yet another set of things to check: if the keyboard works in GRUB, it means that UEFI is probably not handing off the USB controller to the OS anymore. Modern systems need this (Windows, Linux) but older ones don't (FreeDOS etc.).
The physical keyboard works in systemd-boot. As soon as I hit enter to boot the selected option, the keyboard dies (numlock light turns off).
So anything that makes the OS try to get control over USB from the firmware fails. The only reason the KVM works is because it's virtual USB directly from the BMC, so the firmware ignores it. You can validate this if you have a PCIe USB card you can plug in, that one would work.
Yep a 4-port USB 3.0 PCIe card (with 4 separate ASM1042A controllers) works. `lsusb` shows the keyboard, and it functions. Same with flashdrives -- `lsblk` shows them. Unfortunatly I can't use this as a fix however, as the boards only have 2 PCIe slots, which typically have SATA expansion and ConnectX cards installed in them.
As for how to fix it if it turns out to be firmware: it might not be fixable. Older firmware + older kernel might be something that worked, and anything newer is simply not going to work.
This is why I hoped that rolling back the BIOS to the previous version would resolve the issues. But doing that + booting Arch install ISO from 2014 through 2018 still didn't work.
There are a lot of edge cases where it might work in UEFI/CSM/DOS but not anywhere else by the way, including lower speed, broken registers (that 'stick' 1.1 bits so setting 2.0 speeds puts the controller in unknown behaviour), handoff problems due to interrupts being broken etc. To actually know what is really wrong, we'd need an XDP or CCA (if a USB 2 version of the SVTCCA exists) or something like that, and signed software and an NDA from Intel.
It could be as simple as a single bit flip due to a degraded address line...
Kernel and even hardware debugging is probably taking this a bit far. On one hand I'd really like to determine what's broken, just to understand why. But at some point I have to give in and just swallow the cost of replacing the boards. And they are from ~2014 after all. Might be time for an upgrade anyway (though they were performing their duties acceptably).
Other things I randomly tried tonight, that, as expected, didn't help:
- Reseating the RAM
- Booting with a single RAM stick
- Pressing and flexing on the CPU heatsink. (I wonder if is that some of the BGA pins have had solder cracking problems?)
- Changing the IPMI MAC to reset the license -- I found a *cough* keygen *cough* for the IPMI and had activated it so I could perform remote BIOS upgrades. Perhaps this activated some other "USB disable" feature (non I could find in the manual, however)
So, two boards with essentially dead onboard USB. What are the chances?
Maybe I'll upgrade these to something based around AM4 low-end Ryzen.
Server boards are so expensive though... Do I really need IPMI -- I want to say yes, but is it worth the price premium?
And before anyone mentions it, yep, I'm aware of piKVM -- very expensive and Pi's are still hard to come by.
Thanks for your help everyone.
