[solved ] supermicro ipmi / bmc virt cdrom and keyboard stops working with too many usb drives.

lofie

New Member
Jul 12, 2013
10
2
3
[SOLVED : Have a working solution from supermicro via a test bios, see update below posted 16th April
It is a cautionary tail if you try to use a large number of USB drives and hubs with the SM motherboards and still want to enjoy using the bmc/ipmi. ]

---- Original post -----

hi folks, was wandering if anyone else has insight etc into this one.

X11SSH-CTF, Bios 2.5, ipmi 1.60 (latest)
USB disk connected using qnap expander box - TL-R1200C-RP
USB pcie card - qnap QXP-10G2U3A in the pcie 8x slot.

So far all tests done via iKVM

It appears as though too many usb disks attached to a X11SSH-CTF motherboard causes issues :
1) 12 disks attached to the onboard controller prevents the virt cdrom appearing and not avilable for boot. However a power reset makes it appear. (not a big prob)
2) 12 disks attached via a qnap pcie usb card and the virt cdrom never appears.
3) 24 disks attached (12 per port) by either the onboard or the pcie usb card and the keyboard stops working.

Have a support call in with SM but this is proving frustrating.

Im guessing the large number of usb is causing the problem. As simply removing the qnap usb jbod(s) removes the issue.
So far nothing tweaked in the BIOS has resolved anything. Neithing with dual booting, efi or legacy. Or oproms disabled.

Any idea on how I could get virtual cdrom and keyboard working?

No problems to move to SOL, which I expect to solve one problem with the keyboard. But since I can not disable the pcie card there is no work around fro getting to the virtual cdrom which proves useful at times.

My next move is to find some usb hubs and drives to reproduce with qnap gear to highlight this is a USB issue.

Could be a disk spin up issue and just needs a tweak of a timout, but not found anything to tweak.

Have a suspiscion that EFI shell would help debug what he usb setup is as far as the bios sees. But no idea when it comes to efi shell.

As you can see this is a storage setup trying to get as much storage for the money - I work for a non profit.
It's an unusual config, but the 12 bay expander is supported undet ubuntu. The usb expander is so far the best TB per dollar for an expander and it has dual PSU. 12 slots for around 1000 usd. The drives appear as usb drives connected via a usb hub.


Cheers - A

PS if I here anyting from QNAP will update things here - currently trying to escalate beyond the front line support.


And some gory details....

allan@longchen1:~$ lsusb -t

This looks like the qnap controller, 3x usb hubs with 4 disks per hub.
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 3: Dev 9, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 1: Dev 5, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 12, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 7, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 3: Dev 4, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 3: Dev 13, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 1: Dev 8, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 15, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 10, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 1: Dev 11, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 2: Dev 14, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 3: Dev 16, If 0, Class=Mass Storage, Driver=uas, 10000M
|__ Port 4: Dev 17, If 0, Class=Mass Storage, Driver=uas, 10000M

/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 3, If 1, Class=CDC Data, Driver=cdc_acm, 12M
|__ Port 1: Dev 3, If 0, Class=Communications, Driver=cdc_acm, 12M
|__ Port 2: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 5000M

These look like the devices provided by the bmc/ipmi
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
|__ Port 6: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 6: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 14: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 1: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 2: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 480M


allan@longchen1:~$ lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 06)
00:02.0 Display controller: Intel Corporation HD Graphics 630 (rev 04)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:16.1 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1)
00:1c.2 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #3 (rev f1)
00:1c.4 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1)
00:1f.0 ISA bridge: Intel Corporation C236 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
02:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller # QNAP usb3.2 gen 2m confirmed with lspci -v
04:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
06:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
06:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
allan@longchen1:~$ lsscsi
[0:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sda
[0:0:1:0] disk ATA WDC WUH721816AL W120 /dev/sdb
[0:0:2:0] disk ATA WDC WUH721816AL W120 /dev/sdc
[0:0:3:0] disk ATA WDC WUH721816AL W120 /dev/sdd
[0:0:4:0] disk ATA WDC WUH721816AL W120 /dev/sde
[0:0:5:0] disk ATA WDC WUH721816AL W120 /dev/sdg
[0:0:6:0] disk ATA WDC WUH721816AL W120 /dev/sdf
[0:0:7:0] disk ATA WDC WUH721816AL W120 /dev/sdh
[1:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdk
[2:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdv
[5:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdw
[6:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdx
[7:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdy
[8:0:0:0] disk ATA WDC WUH721816AL W120 /dev/sdz
[9:0:0:0] cd/dvd ATEN Virtual CDROM YS0J /dev/sr0
[10:0:0:0] disk HGST HUS QNAP 0 /dev/sdi
[11:0:0:0] disk HGST HUS QNAP 0 /dev/sdj
[12:0:0:0] disk HGST HUS QNAP 0 /dev/sdl
[13:0:0:0] disk HGST HUS QNAP 0 /dev/sdm
[14:0:0:0] disk HGST HUS QNAP 0 /dev/sdn
[15:0:0:0] disk HGST HUS QNAP 0 /dev/sdo
[16:0:0:0] disk HGST HUS QNAP 0 /dev/sdp
[17:0:0:0] disk HGST HUS QNAP 0 /dev/sdq
[18:0:0:0] disk HGST HUS QNAP 0 /dev/sdr
[19:0:0:0] disk HGST HUS QNAP 0 /dev/sds
[20:0:0:0] disk HGST HUS QNAP 0 /dev/sdt
[21:0:0:0] disk HGST HUS QNAP 0 /dev/sdu


lspci -v
...
02:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller (prog-if 30 [XHCI])
Subsystem: QNAP Systems, Inc. ASM2142 USB 3.1 Host Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at df400000 (64-bit, non-prefetchable) [size=32K]
Capabilities: <access denied>
Kernel driver in use: xhci_hcd
...
 
Last edited:

lofie

New Member
Jul 12, 2013
10
2
3
Still no update from Supermicro who want me to send them the jbod so they can test :-(
 
Last edited:

lofie

New Member
Jul 12, 2013
10
2
3
Latest update... after far too much time in the server room some progress.

A test bios from SM has now fixed this problem. Which is great!

Problem is due to too many usb3 hubs. Theres a hard limit in the bios after which the ikvm keyboard stops working. So if you are planning a ghetto usb raid backup and want ipmi for bios + boot test well.

Just want to say this has nothing to do specifically with the qnap jbod. Or to do with using a pcie qnap card. Problem was found using both onboard usb 3 and pcie usb 3 card.

Attaching just one qnap usb 3 jbod with 12 drives can push you past the limit.

However now the ikvm keyboard is working the secondary problem of the virtual cdrom drive not working still remains which appears to be linked to the number of usb attached storage devices. The bios prioritises the jbod drives over any other. And currently no way to disable the pcie usb card afaik.

If there's no workaround will install a recovery partion on both the the internal sata drives, which are mdadm mirrored. Hopefully this will cover most scenarios.

Bottom line - you'll have a lot of problems if you plan to use a large amount of usb drives attched tou your SM motherboard. (X11SSH-CTF)

It was meant to be a budget setup, prioritising cost over performance, but the time cost involved has negated any savings of using a SAS connected jbod. Which is a pity as the TL-1200C-RP qnap jbods are a steal for connecting a lot of disks if you really want a dual psu setup. We were looking for 400TB of presented storage and was even considering using one main controller, with 4x qnap jbods to achieve this. (18tb seagate exos drives raid 1)

Regards - A
 

lofie

New Member
Jul 12, 2013
10
2
3
Final update... Supermicro provided further test bios's.

The ipmi keyboard was related to the number of usb 3 hubs on the system. Witht he new test bios And now the keyboard works flawlessly.

The test bios will theoritically support upto 52 drives available to boot from.

As for booting from the virtual cdrom, the workaround is a combination of power cycle, boot to bios or boot loader, and then power reset or ctrl-alt-del. Then it will appear in the boot menu.
Not smooth but it is a now in a workable state.

Only tested in UEFI boot mode.

Regards - A
 
  • Like
Reactions: lurkusmaximus