Hello everyone, my first post here after years of reading the main site 
I have encountered a truly baffling issue on my new server setup:
The setup: I have two "virtually identical" servers, each with a Supermicro H12SSL-I motherboard, AMD Epyc 7302P, 128GB SKHynix RAM, and crucially, two X520-DA2 NICs. The CPUs are at the same microcode patch level, all motherboard firmware (CPLD, BIOS and IPMI) are up to date and consequently in parity. In total, I own 3 X520-DA2 adapters, one Dell OEM, two Intel. From different vendors and bought at different times.
The issue: when booting either of the two machines, two out of the three X520-DA2 adapters (the Dell and one of the Intels) consistently cause the CPU direct-attached USB 3.0 ports 0 and 1 to stop working after booting into an operating system. It is notable that both of these USB3.0 ports work just fine in the BIOS / EFI shell / GRUB, and only stop working once a kernel is fully booted and running (I have tried both Ubuntu and Debian). All other USB 3.0 ports work fine throughout the duration of the boot process, all the way into interacting with the machine from userland.
My current guess: something about PCIe address space perhaps? I am thinking perhaps the other two cards have an older (or different, somehow incompatible) firmware that perhaps causes system peripherals to fail to initialize. I have monitored lsusb, lspci and dmesg (grepping for USB events, but not finding anything notable, like an initialization failure) with both the working X520-DA2 and the two problematic cards. With the problematic cards, significantly less USB peripherals appear including some that I believe the IPMI uses to talk to the system for OOB management (I haven't gotten a chance to confirm this, in the many hours i have had so far been able to dedicate to power cycling these machines, and waiting through the lengthy Supermicro initialization)
The question: Any ideas what is causing this? TL;DR: one of 3 X520s causes CPU usb to stop working on mutliple systems when installed
Any ideas much appreciated!
I have encountered a truly baffling issue on my new server setup:The setup: I have two "virtually identical" servers, each with a Supermicro H12SSL-I motherboard, AMD Epyc 7302P, 128GB SKHynix RAM, and crucially, two X520-DA2 NICs. The CPUs are at the same microcode patch level, all motherboard firmware (CPLD, BIOS and IPMI) are up to date and consequently in parity. In total, I own 3 X520-DA2 adapters, one Dell OEM, two Intel. From different vendors and bought at different times.
The issue: when booting either of the two machines, two out of the three X520-DA2 adapters (the Dell and one of the Intels) consistently cause the CPU direct-attached USB 3.0 ports 0 and 1 to stop working after booting into an operating system. It is notable that both of these USB3.0 ports work just fine in the BIOS / EFI shell / GRUB, and only stop working once a kernel is fully booted and running (I have tried both Ubuntu and Debian). All other USB 3.0 ports work fine throughout the duration of the boot process, all the way into interacting with the machine from userland.
My current guess: something about PCIe address space perhaps? I am thinking perhaps the other two cards have an older (or different, somehow incompatible) firmware that perhaps causes system peripherals to fail to initialize. I have monitored lsusb, lspci and dmesg (grepping for USB events, but not finding anything notable, like an initialization failure) with both the working X520-DA2 and the two problematic cards. With the problematic cards, significantly less USB peripherals appear including some that I believe the IPMI uses to talk to the system for OOB management (I haven't gotten a chance to confirm this, in the many hours i have had so far been able to dedicate to power cycling these machines, and waiting through the lengthy Supermicro initialization)
The question: Any ideas what is causing this? TL;DR: one of 3 X520s causes CPU usb to stop working on mutliple systems when installed
Any ideas much appreciated!
