Supermicro A1SAM-2550F USB Not Working In OS

[twistedpair]

I have two servers with A1SAM-2550F motherboards. These are low-power SoC Atom-based boards. They both run recent builds of ArchLinux.

For the longest time I have been using the built-in IPMI KVM to do any remote control that I can't do over SSH. These servers haven't had a keyboard or other USB devices connected for a long time. Recently however I wanted to use one of the servers for some home automation stuff -- ZigBee via a USB stick. This is when I discovered that Linux wouldn't detect any USB device except for the IPMI KVM's virtual devices.

What is weird though is that USB devices are detected and functional in UEFI -- I can use a USB keyboard to navigate the BIOS menus and type in the EFI shell, and USB flash drives are detected and I can boot from them. It seems that once the OS boots the onboard USB breaks. I can even observe this with the keyboard numlock LED -- it remains illuminated until the OS boots.

I've tried the following troubleshooting steps with no luck:

• Enable/disable EHCI handoff
• Enable/disable Legacy USB support.
• Restoring BIOS settings to defaults (both via the menu and clear CMOS jumper).
• Rollback to a older BIOS.
• Resetting the IMPI settings.
• Checking if there is a "disable USB" jumper.
• Booting various Linux distro live ISOs, and older versions from a few years ago for older kernels.
• Booting OS's in BIOS/CSM mode.
• Installing Windows 10 and the drivers from the Supermicro website.

When booting a Windows 10 Install USB, an attached physical USB keyboard isn't recognized in Windows Setup. When attempting an an install, the Windows Setup program cannot find the install media (on USB) and prompts for a missing driver.

The USB controller does show up in `lspci`. There doesn't seem to be anything obvious in `dmesg` (grepping for echi).

I know that USB on these boards has worked in the past. I'm uncertain exactly as to when it stopped working.

I'm aware that the Avoton Atoms do have a known issue where the board eventually stops booting. I had one fail already, replaced under RMA. This isn't the same thing.

I even reached out to Supermicro support and they have no idea.

Any thoughts?

 

[ericloewe]

Enable/disable EHCI handoff

What about XHCI handoff? Grasping at straws, but that's a weird one...

This is when I discovered that Linux wouldn't detect any USB device except for the IPMI KVM's virtual devices.

But do they work?

 

[zac1]

Process of elimination: does your USB keyboard work in some kind of live boot environment?

Whoops. Yeah, that's a weird one.

 

[ericloewe]

Also: The USB 3.0 ports are from a separate Renesas controller. Is the behavior the same on USB 3.0 and USB 2.0 ports?

 

[RolloZ170]

this remembers me to the first intel 100 series chipset & windows 7 issues. no OS driver for the new PCH USB ports,
from a point the OS should take over the USB the connections was lost.
was there a similar the first with Ryzen chipset ?

 

[twistedpair]

What about XHCI handoff? Grasping at straws, but that's a weird one...

Board only has USB 2.0 onboard. So no XHCI options in the BIOS.

Here are the Advanced -> Chipset Configuration -> I/O Fabric options in the BIOS:



Also note the USB Devices text at the top. I have 1 physical keyboard connected and 1 flash drive. Combined with the IPMI KVM nets the detected 1 flash drive, 2 keyboard, 1 mouse.

But do they work?

Yes, the IPMI virtual devices do work, both in the BIOS and once an OS has booted. Kayboard and mouse both work in the OS, and I can mount an ISO no problem using the virtual CDROM drive. This is partly why I never noticed that physical USB wasn't working. For most things I ssh into the box, but on the rare occasion that I need to interact with the machine the IPMI KVM works.

Process of elimination: does your USB keyboard work in some kind of live boot environment?

Whoops. Yeah, that's a weird one.

Yep, physical USB stuff works in the firmware/BIOS. UEFI shell can list the contents of USB flash drives, for example.
However, if I boot from a live USB, once the OS has started, the physical USB stops working. For linux this doesn't cause major issues since the whole rootfs is loaded into RAM. But for the Windows 10 setup the result is that the installer cannot find its own filesystem and has issues. Older Windows installers (7, 8) actually blue screen.

Also: The USB 3.0 ports are from a separate Renesas controller. Is the behavior the same on USB 3.0 and USB 2.0 ports?

There is no onboard USB 3.0.

this remembers me to the first intel 100 series chipset & windows 7 issues. no OS driver for the new PCH USB ports,
from a point the OS should take over the USB the connections was lost.
was there a similar the first with Ryzen chipset ?

This board is Intel Atom. It's also pretty old at this point (Avoton was introduced ~2013). Given its age, drivers for the most of the onboard devices are present in Windows 10 and current Linux. On a fresh Windows 10 install only a few board specific SMBUS devices are listed in Device Manager as unknown. Under Linux, `lspci` detects everything.


Had access to one of the boxes again today and was able to boot into Linux and get some info:

$ lspci
00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 02)
00:17.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA2 Controller (rev 02)
00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 02)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
03:00.0 SATA controller: ASMedia Technology Inc. Device 1164 (rev 02)

$ sudo lspci -v -s 00:16.0
00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 02) (prog-if 20 [EHCI])
        Subsystem: Super Micro Computer Inc Device 0820
        Flags: bus master, medium devsel, latency 0, IRQ 23
        Memory at df2f3000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
        Kernel driver in use: ehci-pci

$ sudo dmesg | grep -i ehci
[    0.586705] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.586713] ehci-pci: EHCI PCI platform driver
[    0.586948] ehci-pci 0000:00:16.0: EHCI Host Controller
[    0.586962] ehci-pci 0000:00:16.0: new USB bus registered, assigned bus number 1
[    0.586980] ehci-pci 0000:00:16.0: debug port 2
[    0.590937] ehci-pci 0000:00:16.0: irq 23, io mem 0xdf2f3000
[    0.603780] ehci-pci 0000:00:16.0: USB 2.0 started, EHCI 1.00
[    0.603908] usb usb1: Product: EHCI Host Controller
[    0.603911] usb usb1: Manufacturer: Linux 5.19.12-arch1-1 ehci_hcd
[    0.853782] usb 1-1: new high-speed USB device number 2 using ehci-pci
[    1.290443] usb 1-1.3: new high-speed USB device number 3 using ehci-pci
[    1.687108] usb 1-1.3.1: new low-speed USB device number 4 using ehci-pci

$ sudo dmesg | grep -i usb
[    0.330898] ACPI: USB4 _OSC: OS supports USB3+ DisplayPort+ PCIe+ XDomain+
[    0.330907] ACPI: USB4 _OSC: OS controls USB3+ DisplayPort+ PCIe+ XDomain+
[    0.363617] ACPI: bus type USB registered
[    0.363648] usbcore: registered new interface driver usbfs
[    0.363668] usbcore: registered new interface driver hub
[    0.363684] usbcore: registered new device driver usb
[    0.403914] pci 0000:00:16.0: quirk_usb_early_handoff+0x0/0x750 took 13733 usecs
[    0.586705] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.586962] ehci-pci 0000:00:16.0: new USB bus registered, assigned bus number 1
[    0.603780] ehci-pci 0000:00:16.0: USB 2.0 started, EHCI 1.00
[    0.603898] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.19
[    0.603904] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.603908] usb usb1: Product: EHCI Host Controller
[    0.603911] usb usb1: Manufacturer: Linux 5.19.12-arch1-1 ehci_hcd
[    0.603914] usb usb1: SerialNumber: 0000:00:16.0
[    0.604155] hub 1-0:1.0: USB hub found
[    0.604571] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.604603] uhci_hcd: USB Universal Host Controller Interface driver
[    0.604686] usbcore: registered new interface driver usbserial_generic
[    0.604698] usbserial: USB Serial support registered for generic
[    0.853782] usb 1-1: new high-speed USB device number 2 using ehci-pci
[    1.010872] usb 1-1: New USB device found, idVendor=8087, idProduct=07db, bcdDevice= 0.02
[    1.010880] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    1.011169] hub 1-1:1.0: USB hub found
[    1.290443] usb 1-1.3: new high-speed USB device number 3 using ehci-pci
[    1.387484] usb 1-1.3: New USB device found, idVendor=0557, idProduct=7000, bcdDevice= 0.00
[    1.387491] usb 1-1.3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    1.387775] hub 1-1.3:1.0: USB hub found
[    1.687108] usb 1-1.3.1: new low-speed USB device number 4 using ehci-pci
[    1.811266] usb 1-1.3.1: New USB device found, idVendor=0557, idProduct=2419, bcdDevice= 1.00
[    1.811272] usb 1-1.3.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    7.555111] input: HID 0557:2419 as /devices/pci0000:00/0000:00:16.0/usb1/1-1/1-1.3/1-1.3.1/1-1.3.1:1.0/0003:0557:2419.0001/input/input1
[    7.614128] hid-generic 0003:0557:2419.0001: input,hidraw0: USB HID v1.00 Keyboard [HID 0557:2419] on usb-0000:00:16.0-1.3.1/input0
[    7.614922] input: HID 0557:2419 as /devices/pci0000:00/0000:00:16.0/usb1/1-1/1-1.3/1-1.3.1/1-1.3.1:1.1/0003:0557:2419.0002/input/input2
[    7.615101] hid-generic 0003:0557:2419.0002: input,hidraw1: USB HID v1.00 Mouse [HID 0557:2419] on usb-0000:00:16.0-1.3.1/input1
[    7.615174] usbcore: registered new interface driver usbhid
[    7.615178] usbhid: USB HID core driver

Given that both Windows 10 and Linux don't detect physical USB devices, I want to label this as hardware failure. However, the fact that physical USB devices still work in the firmware throws my confidence off here.

 

[zac1]

Is VT-d enabled? Saw a similar issue resolved by enabling IOMMU.

BIOS up-to-date? Try resetting BIOS?

And long shot, but it saved my butt when I thought I got a DOA board... remove CMOS battery for a short while?

 

[twistedpair]

Is VT-d enabled? Saw a similar issue resolved by enabling IOMMU.

I believe so:



There isn't an explicit option for IOMMU in the BIOS that I can find.

I'm pretty sure I stumbled upon that stack overflow thread previously and tried a bunch of the suggestions. But I'll try the Linux kernel options (such as `iommu=soft`) thrown around in the comments again.

BIOS up-to-date? Try resetting BIOS?

I was running the latest available. Updating the BIOS and IPMI BMC firmware was the only thing I could think of that I'd changed at the hardware level for a long time. I reached out to Supermicro support for an older BIOS, which they provided. It made no difference.

SM support also mentioned that they haven't received any other reports of USB issues like this.

I've also reset the BIOS settings via the menu and by shorting the CLRCMOS jumper with the battery removed. No dice.

And long shot, but it saved my butt when I thought I got a DOA board... remove CMOS battery for a short while?

Worth a shot!

I might even try re-flashing the BIOS via USB rather than via IPMI to see if that makes a difference.


Edit: Yes, looks like VT-x is enabled. Booting with `iommu=soft` make no difference.

$ lscpu
...
Virtualization features:
  Virtualization:        VT-x

 

[twistedpair]

Though I guess VT-x and VT-d are different. One is hardware virtualisation, the other is device passthrough/iommu.

 

[twistedpair]

(Re-)Flashed the latest BIOS via FreeDOS + AFUDOS.exe. No change.

One thing that's interesting is that the physical keyboard continues to function once FreeDOS has booted.
I assume FreeDOS is using UEFI CSM, and hence the keyboard BIOS stuff is being emulated by UEFI, rather than the USB stack being controlled by FreeDOS.

Also removed the CMOS battery for 10 minutes with the power disconnected. Didn't help either

 

[zac1]

(Re-)Flashed the latest BIOS via FreeDOS + AFUDOS.exe. No change.

One thing that's interesting is that the physical keyboard continues to function once FreeDOS has booted.
I assume FreeDOS is using UEFI CSM, and hence the keyboard BIOS stuff is being emulated by UEFI, rather than the USB stack being controlled by FreeDOS.

Also removed the CMOS battery for 10 minutes with the power disconnected. Didn't help either

Have you tried live booting both UEFI and BIOS?

 

[twistedpair]

Tried booting the latest ArchLinux install ISO (archlinux-2022.10.01-x86_64.iso), written to a USB flash drive using Rufus 3.20 in MBR/BIOS+UEFI mode.

Using BIOS defaults.

Booting the install ISO in BIOS mode still results in physical USB devices not working (ie, can't type in the initramfs shell prompt using the attached keyboard -- KVM keyboard works fine)

In either mode, the OS doesn't boot as the kernel is unable to locate the rootfs because the USB device can't be found:

 

[zac1]

Have you reset BIOS yet?

 

[twistedpair]

Many times. Using both the onboard CLRCMOS jumper and the Restore Defaults option in the setup menu.

 

[zac1]

Many times. Using both the onboard CLRCMOS jumper and the Restore Defaults option in the setup menu.

Ah, duh. Second time I've overlooked or spaced a detail you already shared, sorry again. Color me ignorant and stumped. I will keep an eye on your thread in hopes of an update and lesson for the future. Best of luck!

 

[twistedpair]

Ah, duh. Second time I've overlooked or spaced a detail you already shared, sorry again. Color me ignorant and stumped. I will keep an eye on your thread in hopes of an update and lesson for the future. Best of luck!

I appreciate your help. I'll post here if I find anything.

 

[zir_blazer]

VT-x and VT-d are different things, and most likely that old Atom platform doesn't support the latter.

Could it be possible than it is actually a buggy Driver or unknown errata that slipped by in both Linux Kernel and Windows? Can you use an era-accurate OS (Some Linux LiveCD from 2014 or so)?
Unless someone has an identical Motherboard and can confirm it works in theirs to point out to some kind of Hardware damage, which as stated before, seems weird because it works in Firmware... Did you tested on both servers?

 

[i386]

did you use the usb ports on the rear or did you connect via cable from the usb headers to a front panel?

 

[RolloZ170]

maybe is it quiet normal the physical USB becomes unavail. if KVM is online ?
have you checked the behaiviour without IPMI KVM enabled ?

 

[twistedpair]

Could it be possible than it is actually a buggy Driver or unknown errata that slipped by in both Linux Kernel and Windows? Can you use an era-accurate OS (Some Linux LiveCD from 2014 or so)?

Unless someone has an identical Motherboard and can confirm it works in theirs to point out to some kind of Hardware damage, which as stated before, seems weird because it works in Firmware... Did you tested on both servers?

A good test to try. I wrote the ArchLinux 2014.01.05 iso to a USB flash drive using Rufus. It failed to mount the rootfs, just like the more recent 2022.10.01 iso.

Tested on both boards with BIOS reset to defaults. Both exhibited the same result:

did you use the usb ports on the rear or did you connect via cable from the usb headers to a front panel?

Rear IO panel USB ports that are soldered directly on the board. Tried different ones too.