Remember, it is the IPMI/BMC that's doing this. They continue to run even when the system is "off" - look at the heartbeat LED on the motherboard.It's now running 'right' again. It required me to completely remove the power plug and let it sit then plug it back in and power up. Rebooting, and powering it off and back on did not fix it. This seems like a glitch in the SuperMicro BIOS -- for not resetting on power-off.
There is probably some provision to cold-reset them on a running system, as long as they are still responding. If they aren't responding, then they will need a power cycle to get them rebooted.