[SOLVED : Have a working solution from supermicro via a test bios, see update below posted 16th April
It is a cautionary tail if you try to use a large number of USB drives and hubs with the SM motherboards and still want to enjoy using the bmc/ipmi. ]
---- Original post -----
hi folks, was wandering if anyone else has insight etc into this one.
X11SSH-CTF, Bios 2.5, ipmi 1.60 (latest)
USB disk connected using qnap expander box - TL-R1200C-RP
USB pcie card - qnap QXP-10G2U3A in the pcie 8x slot.
So far all tests done via iKVM
It appears as though too many usb disks attached to a X11SSH-CTF motherboard causes issues :
1) 12 disks attached to the onboard controller prevents the virt cdrom appearing and not avilable for boot. However a power reset makes it appear. (not a big prob)
2) 12 disks attached via a qnap pcie usb card and the virt cdrom never appears.
3) 24 disks attached (12 per port) by either the onboard or the pcie usb card and the keyboard stops working.
Have a support call in with SM but this is proving frustrating.
Im guessing the large number of usb is causing the problem. As simply removing the qnap usb jbod(s) removes the issue.
So far nothing tweaked in the BIOS has resolved anything. Neithing with dual booting, efi or legacy. Or oproms disabled.
Any idea on how I could get virtual cdrom and keyboard working?
No problems to move to SOL, which I expect to solve one problem with the keyboard. But since I can not disable the pcie card there is no work around fro getting to the virtual cdrom which proves useful at times.
My next move is to find some usb hubs and drives to reproduce with qnap gear to highlight this is a USB issue.
Could be a disk spin up issue and just needs a tweak of a timout, but not found anything to tweak.
Have a suspiscion that EFI shell would help debug what he usb setup is as far as the bios sees. But no idea when it comes to efi shell.
As you can see this is a storage setup trying to get as much storage for the money - I work for a non profit.
It's an unusual config, but the 12 bay expander is supported undet ubuntu. The usb expander is so far the best TB per dollar for an expander and it has dual PSU. 12 slots for around 1000 usd. The drives appear as usb drives connected via a usb hub.
Cheers - A
PS if I here anyting from QNAP will update things here - currently trying to escalate beyond the front line support.
And some gory details....
allan@longchen1:~$ lsusb -t
This looks like the qnap controller, 3x usb hubs with 4 disks per hub.
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 3: Dev 9, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 1: Dev 5, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 12, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 7, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 3: Dev 4, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 3: Dev 13, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 1: Dev 8, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 15, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 10, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 1: Dev 11, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 14, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 3: Dev 16, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 17, If 0, Class=Mass Storage, Driver=uas, 10000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 3, If 1, Class=CDC Data, Driver=cdc_acm, 12M
|__ Port 1: Dev 3, If 0, Class=Communications, Driver=cdc_acm, 12M
|__ Port 2: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 5000M
These look like the devices provided by the bmc/ipmi
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
|__ Port 6: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 6: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 14: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 1: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 2: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 480M
allan@longchen1:~$ lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 06)
00:02.0 Display controller: Intel Corporation HD Graphics 630 (rev 04)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:16.1 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1)
00:1c.2 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #3 (rev f1)
00:1c.4 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1)
00:1f.0 ISA bridge: Intel Corporation C236 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
02:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller # QNAP usb3.2 gen 2m confirmed with lspci -v
04:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
06:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
06:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
allan@longchen1:~$ lsscsi
[0:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sda
[0:0:1:0] disk ATA WDC WUH721816AL W120 /dev/sdb
[0:0:2:0] disk ATA WDC WUH721816AL W120 /dev/sdc
[0:0:3:0] disk ATA WDC WUH721816AL W120 /dev/sdd
[0:0:4:0] disk ATA WDC WUH721816AL W120 /dev/sde
[0:0:5:0] disk ATA WDC WUH721816AL W120 /dev/sdg
[0:0:6:0] disk ATA WDC WUH721816AL W120 /dev/sdf
[0:0:7:0] disk ATA WDC WUH721816AL W120 /dev/sdh
[1:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdk
[2:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdv
[5:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdw
[6:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdx
[7:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdy
[8:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdz
[9:0:0:0] cd/dvd ATEN Virtual CDROM YS0J /dev/sr0
[10:0:0:0] disk HGST HUS QNAP 0 /dev/sdi
[11:0:0:0] disk HGST HUS QNAP 0 /dev/sdj
[12:0:0:0] disk HGST HUS QNAP 0 /dev/sdl
[13:0:0:0] disk HGST HUS QNAP 0 /dev/sdm
[14:0:0:0] disk HGST HUS QNAP 0 /dev/sdn
[15:0:0:0] disk HGST HUS QNAP 0 /dev/sdo
[16:0:0:0] disk HGST HUS QNAP 0 /dev/sdp
[17:0:0:0] disk HGST HUS QNAP 0 /dev/sdq
[18:0:0:0] disk HGST HUS QNAP 0 /dev/sdr
[19:0:0:0] disk HGST HUS QNAP 0 /dev/sds
[20:0:0:0] disk HGST HUS QNAP 0 /dev/sdt
[21:0:0:0] disk HGST HUS QNAP 0 /dev/sdu
lspci -v
...
02:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller (prog-if 30 [XHCI])
Subsystem: QNAP Systems, Inc. ASM2142 USB 3.1 Host Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at df400000 (64-bit, non-prefetchable) [size=32K]
Capabilities: <access denied>
Kernel driver in use: xhci_hcd
...
It is a cautionary tail if you try to use a large number of USB drives and hubs with the SM motherboards and still want to enjoy using the bmc/ipmi. ]
---- Original post -----
hi folks, was wandering if anyone else has insight etc into this one.
X11SSH-CTF, Bios 2.5, ipmi 1.60 (latest)
USB disk connected using qnap expander box - TL-R1200C-RP
USB pcie card - qnap QXP-10G2U3A in the pcie 8x slot.
So far all tests done via iKVM
It appears as though too many usb disks attached to a X11SSH-CTF motherboard causes issues :
1) 12 disks attached to the onboard controller prevents the virt cdrom appearing and not avilable for boot. However a power reset makes it appear. (not a big prob)
2) 12 disks attached via a qnap pcie usb card and the virt cdrom never appears.
3) 24 disks attached (12 per port) by either the onboard or the pcie usb card and the keyboard stops working.
Have a support call in with SM but this is proving frustrating.
Im guessing the large number of usb is causing the problem. As simply removing the qnap usb jbod(s) removes the issue.
So far nothing tweaked in the BIOS has resolved anything. Neithing with dual booting, efi or legacy. Or oproms disabled.
Any idea on how I could get virtual cdrom and keyboard working?
No problems to move to SOL, which I expect to solve one problem with the keyboard. But since I can not disable the pcie card there is no work around fro getting to the virtual cdrom which proves useful at times.
My next move is to find some usb hubs and drives to reproduce with qnap gear to highlight this is a USB issue.
Could be a disk spin up issue and just needs a tweak of a timout, but not found anything to tweak.
Have a suspiscion that EFI shell would help debug what he usb setup is as far as the bios sees. But no idea when it comes to efi shell.
As you can see this is a storage setup trying to get as much storage for the money - I work for a non profit.
It's an unusual config, but the 12 bay expander is supported undet ubuntu. The usb expander is so far the best TB per dollar for an expander and it has dual PSU. 12 slots for around 1000 usd. The drives appear as usb drives connected via a usb hub.
Cheers - A
PS if I here anyting from QNAP will update things here - currently trying to escalate beyond the front line support.
And some gory details....
allan@longchen1:~$ lsusb -t
This looks like the qnap controller, 3x usb hubs with 4 disks per hub.
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 3: Dev 9, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 1: Dev 5, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 12, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 7, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 3: Dev 4, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 3: Dev 13, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 1: Dev 8, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 15, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 10, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 1: Dev 11, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 14, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 3: Dev 16, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 17, If 0, Class=Mass Storage, Driver=uas, 10000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 3, If 1, Class=CDC Data, Driver=cdc_acm, 12M
|__ Port 1: Dev 3, If 0, Class=Communications, Driver=cdc_acm, 12M
|__ Port 2: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 5000M
These look like the devices provided by the bmc/ipmi
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
|__ Port 6: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 6: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 14: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 1: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 2: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 480M
allan@longchen1:~$ lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 06)
00:02.0 Display controller: Intel Corporation HD Graphics 630 (rev 04)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:16.1 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1)
00:1c.2 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #3 (rev f1)
00:1c.4 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1)
00:1f.0 ISA bridge: Intel Corporation C236 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
02:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller # QNAP usb3.2 gen 2m confirmed with lspci -v
04:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
06:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
06:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
allan@longchen1:~$ lsscsi
[0:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sda
[0:0:1:0] disk ATA WDC WUH721816AL W120 /dev/sdb
[0:0:2:0] disk ATA WDC WUH721816AL W120 /dev/sdc
[0:0:3:0] disk ATA WDC WUH721816AL W120 /dev/sdd
[0:0:4:0] disk ATA WDC WUH721816AL W120 /dev/sde
[0:0:5:0] disk ATA WDC WUH721816AL W120 /dev/sdg
[0:0:6:0] disk ATA WDC WUH721816AL W120 /dev/sdf
[0:0:7:0] disk ATA WDC WUH721816AL W120 /dev/sdh
[1:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdk
[2:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdv
[5:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdw
[6:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdx
[7:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdy
[8:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdz
[9:0:0:0] cd/dvd ATEN Virtual CDROM YS0J /dev/sr0
[10:0:0:0] disk HGST HUS QNAP 0 /dev/sdi
[11:0:0:0] disk HGST HUS QNAP 0 /dev/sdj
[12:0:0:0] disk HGST HUS QNAP 0 /dev/sdl
[13:0:0:0] disk HGST HUS QNAP 0 /dev/sdm
[14:0:0:0] disk HGST HUS QNAP 0 /dev/sdn
[15:0:0:0] disk HGST HUS QNAP 0 /dev/sdo
[16:0:0:0] disk HGST HUS QNAP 0 /dev/sdp
[17:0:0:0] disk HGST HUS QNAP 0 /dev/sdq
[18:0:0:0] disk HGST HUS QNAP 0 /dev/sdr
[19:0:0:0] disk HGST HUS QNAP 0 /dev/sds
[20:0:0:0] disk HGST HUS QNAP 0 /dev/sdt
[21:0:0:0] disk HGST HUS QNAP 0 /dev/sdu
lspci -v
...
02:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller (prog-if 30 [XHCI])
Subsystem: QNAP Systems, Inc. ASM2142 USB 3.1 Host Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at df400000 (64-bit, non-prefetchable) [size=32K]
Capabilities: <access denied>
Kernel driver in use: xhci_hcd
...
Last edited: