I believe this problem likely turns out to be an ESXI bug with message-signaled interrupts that was exposed by a change to the linux-x86 interrupt handling code post kernel 4.1.
There is a thread discussing the issue on the linux-scsi mailing list:
http://www.spinics.net/lists/linux-scsi/msg95506.html
If you follow the links in that message there is a description (quoted below.) The claim is that it will be fixed in ESXI 6.0p3/5.5p8 mid-year.
The workaround that is working for me now is to add "mpt2sas.msix_disable=1" to the kernel command-line. Note that for 4.4 and later kernels, I think you will need to use "mpt3sas.msix_disable=1" as the mpt2/3 drivers are merged at that point. Actually, you could add both and nothing bad will happen.
Note that there is some odd behavior to this bug. The problem seems to be related to how the interrupt logic is reinitialized (or not reinitialized) on VM reboots. So if the previous reboot (different kernel, command-line options) left the HW in a "good" state you might not see the issue on the next boot. But then subsequent boots will fail.
For what it's worth, I am seeing this on a Fedora install. I use a custom configured kernel with the drivers linked directly in, so it's not a systemd/module issue. I am using P19 LSI firmware on LSI 9211 cards.
There is a thread discussing the issue on the linux-scsi mailing list:
http://www.spinics.net/lists/linux-scsi/msg95506.html
If you follow the links in that message there is a description (quoted below.) The claim is that it will be fixed in ESXI 6.0p3/5.5p8 mid-year.
The workaround that is working for me now is to add "mpt2sas.msix_disable=1" to the kernel command-line. Note that for 4.4 and later kernels, I think you will need to use "mpt3sas.msix_disable=1" as the mpt2/3 drivers are merged at that point. Actually, you could add both and nothing bad will happen.
Note that there is some odd behavior to this bug. The problem seems to be related to how the interrupt logic is reinitialized (or not reinitialized) on VM reboots. So if the previous reboot (different kernel, command-line options) left the HW in a "good" state you might not see the issue on the next boot. But then subsequent boots will fail.
For what it's worth, I am seeing this on a Fedora install. I use a custom configured kernel with the drivers linked directly in, so it's not a systemd/module issue. I am using P19 LSI firmware on LSI 9211 cards.
List: linux-kernel
Subject: Re: VMware PCI passthrough regression
From: Thomas Gleixner <tglx () linutronix ! de>
Date: 2016-01-14 21:15:59
Jason,
On Thu, 14 Jan 2016, Jason Taylor wrote:
> I've run into a regression using PCI passthrough with the 4.4
> kernel.
Actually that is a 4.2 kernel according to the dmesg in the bug tracker.
> Attempting to passthrough an LSI card with ESXi version 6. Seeing
> timeouts and oops in the log and the disks do not show up.
The timeouts are probably related to missing irq delivery, so it might be
related to the big overhaul of the x86 irq subsystem.
The oops is a genuine driver bug probably unearthed by the irq issue. That
should be reported seperately to the megasas folks.
> I performed a bisect which tracked down the issue to the commit below.
>
> More details are available in a bug report I filed with Ubuntu:
> Bug #1528849 “PCI passthrough of LSI card fails” : Bugs : linux package : Ubuntu
>
> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c
> x86/MSI: Use hierarchical irqdomains to manage MSI interrupts
I have no idea how that breaks the vmware passthrough. Can you please verify
- whether that kernel works on the real hardware with that LSI card
- whether that kernel works in a KVM guest with that card passed through
If one of those things break, we can certainly help to analyse that. If not,
then you need to talk to vmware.