LSI sas2 and sas3 HBA cards - optimal firmware versions, features

[sfbayzfs]

After general wisdom of "flash the latest LSI firmware release" failed me, I am trying to determine the best versions of LSI IT mode firmware for their SAS2 and SAS3 cards based on 2008, 2308, and 3008 (and multiport version) chipsets. The IR mode firmware, while runnable on many of these cards, seems to be on a different release cycle and codebase with it's own quirks and bugs, so although peripherally interesting, it is not the core of what I am trying to figure out here.

Results Summary so far:

Generally, match firmware to your OS's provided LSI driver version

mpt2sas:

Solaris and variants: unknown, but no problems reported with P19
FreeBSD 9.x: P16
FreeBSD 10.x: P16
RHEL/CentOS 6.x: P16
RHEL/CentOS 7: P18
Fedora 20: P16

mpt3sas:
Centos 7: P4, latest card firmware is P7, no known problems.

For LSI SAS2 cards, P16 mpt2sas drivers ship with FreeBSD 9 and 10, Centos 6, Fedora 20, and probably many others. P19 firmware is reported to work well in Linux if you install the P19 drivers yourself, but P19 firmware with P16 drivers has had problems reported in FreeBSD.

For SAS3 cards, they seem to have started the "phase" system over again and are on something like P6 last time I checked - does anyone know what the name of the driver is for these, and or the shipping version with FreeBSD, Centos, Ubuntu, Solaris, etc.?

History:

My inBack in December, I ran into what turned out to be the horrible Avago / LSI P20 firmware problems -when setting up a recently updated IBM M1015 with FreeNAS - writes were fine, but reads would register a stream of CRC errors on all of the drives and slowed to a crawl. After much research, it appeared that P19 was the last good firmware from LSI, and P20 was useless for anyone not using the card in Windows.

I also learned that FreeBSD 9 and 10 seem to include the P16 driver, and although rare, I did find at least 1 report of problems with P19 firmware in FreeBSD with the P16 driver, which rolling back to P16 firmware fixed.

The firmware release notes from LSI are less detailed than I would like, does anyone have any experience with particular issues with particular firmware/driver versions being fixed (or not fixed) A common claim of new releases is improved compatibility with SAS expanders and backplanes, but does anyone have evidence of problems or proper operation with any particular firmware/driver version combinations? It seems that SAS1 expanders/backplanes are often tricky, although most of the problem reports I have seen for them are with IR mode firmware, not HBA firmware - has anyone had issues with SAS1 expanders?

My question to you:

Or another way to think about this is - has anyone had drive/expander problems with P16 IT mode firmware which were fixed by going to another version of the firmware/drivers, and if so, what are the details of your situation? Has anyone had problems with HP, LSI, Chenbro or other SAS1 expanders, or JBODs like the Rackables/SGI SE3016 units or others?

 

[gea]

I can only comment on Solaris where you can use the newest firmware
- but should not be the first when it comes out as newer software may contain bugs,
just like we see it with P20 firmware on any platform especially with fast SSDs.

The common suggestion on Solaris SAS2 is: use P19 firmware
A good rule is to follow maillists like OmniOS-discuss (for OmniOS) where
problems are reported and discusses quite immediately.

For SAS3, I have not seen any reports but if you do not need I would prefer the proven
SAS2 controllers like LSI 9207 (use SAS3 only with expander + many disks behind).

Expander use should work mostly but there are problem reports where a failed Sata disk
can block the whole expander. Professional storage vendors (not only ZFS related ones)
mostly deny support for expander + Sata disk solutions at all or with disks that are not
delivered and tested by the vendor itself (sometimes with a modified firmware).

I would always prefer SAS disks + expander or multiple HBA + Sata disks solutions.
ex [OmniOS-discuss] iSCSI traffic suddenly comes to a halt and then resumes
and following

 

[TuxDude]

I've never had an issue with my setup - perfectly stable and good performance. I've got a M1015 running firmware P18 (I received it that way) and did nothing but plug it all in and let it go. With this discussion on matching driver/firmware versions I just did a quick check, and the OS-provided driver is also version 18 for me, nice coincidence. I'm running Gentoo with linux kernel 3.18.7 (gentoo-sources, though I'd assume that this driver is the same in vanilla source). Maybe later I'll go through the commit history for the linux kernel and make a table of which kernel versions provide which LSI drivers. It's easy enough to find in the kernel source, under drivers/scsi/mpt2sas/mpt2sas_base.h

 

[gea]

If you buy a new card you usually get the newest firmware.
In the last 5 years this was never a problem for me.

I mostly flashed the firmware to have the same firmware
because I usually use more than one HBA.

P20 was the first that should be avoided.

 

[TuxDude]

Ok, so I've gathered some data both from a few machines I have easy access to, as well as the git history at kernel.org, and have compiled the list of built-in driver versions.

Keep in mind that distributions can and do modify the vanilla linux kernel before they make it available to end users. For example RedHat will never update the kernel version of a RHEL release - RHEL6.0 (and clones) was released with a 2.6.32 kernel, is still running 2.6.32 today at RHEL6.6, and will still be running 2.6.32 for all the remaining years until it is end-of-life. But RedHat has back-ported the newer LSI drivers back to the older kernel version, so RHEL6.6 gets MPT2SAS version 16 instead of the version 2 that the upstream kernel includes. With that in mind, first up is the list from kernel.org - the first column is the linux kernel version, second column is MPT2SAS driver version. I have only included lines where the driver version changed, so eg. kernel 3.12 came with driver 16, kernels 3.13-3.17 are all also on driver 16.

4.0 20.100.00.00
3.18 18.100.00.00
3.12 16.100.00.00
3.11 15.100.00.00
3.7 14.100.00.00
3.5 13.100.00.00
3.3 12.100.00.00
3.2 10.100.00.00
3.1 09.100.00.00
3.0 08.100.00.02
2.6.39 08.100.00.00
2.6.38 07.100.00.00
2.6.36 06.100.00.00
2.6.35 05.100.00.02
2.6.34 04.100.01.00
2.6.33 03.100.03.00
2.6.32 02.100.03.00
2.6.31 01.100.04.00
2.6.30 01.100.02.00

2.6.30 was the first linux kernel to include the MPT2SAS driver - 2.6.29 is missing the entire directory where the files would be located.

And here is the much shorter list of distributions that I checked. First column is distribution, then the currently-running kernel version, and last the MPT2SAS driver version.

Fedora 20 Kernel 3.17.7 MPT2SAS 16.100.00.00
CentOS 6.6 Kernel 2.6.32 MPT2SAS 16.101.00.00
CentOS 7.1 Kernel 3.10.0 MPT2SAS 18.100.00.00

 

[sfbayzfs]

4.0 20.100.00.00

Hmmm, scary

Fedora 20 Kernel 3.17.7 MPT2SAS 16.100.00.00
CentOS 6.6 Kernel 2.6.32 MPT2SAS 16.101.00.00
CentOS 7.1 Kernel 3.10.0 MPT2SAS 18.100.00.00

Thanks for doing the legwork for these, I have updated the recommendations list at the top. Whatever command I used in Centos 7 claimed a P16 driver, but it could be that the version string was not updated with the backport (as you mentioned, Redhat backports everything under the same version with different release numbers in their RPMs), in any case I see version 18 running on it now:
[ 0.893676] mpt2sas version 18.100.00.00 loaded

 

[sfbayzfs]

I have updated the info above with your Solaris recommendation, but what version is that, and do you know when the P19 drivers were added to Solaris/Openindiana?

Expander use should work mostly but there are problem reports where a failed Sata disk
can block the whole expander. Professional storage vendors (not only ZFS related ones)
mostly deny support for expander + Sata disk solutions at all or with disks that are not
delivered and tested by the vendor itself (sometimes with a modified firmware).

I would always prefer SAS disks + expander or multiple HBA + Sata disks solutions.
ex [OmniOS-discuss] iSCSI traffic suddenly comes to a halt and then resumes
and following

That link didn't work for me, but I found it by googling for the subject. Unfortunately I already have my disks (Hitachi/HGST low power 4TB) and I can't afford SAS disks for this, so I'm sticking with them.

I do have a few failed drives lying around - I will put them in my Supermicro SAS2 expander backplanes and see what happens!

I also have a couple of chassis with the older SAS1 versions of the same backplanes, I was planning to move my disks over to that chassis got a bandwidth test (I have been doing various ZOL tests on my new Supermicro SC847 based system in the builds forum) but I will do the failed disk test as well.

 

[TuxDude]

Whatever command I used in Centos 7 claimed a P16 driver, but it could be that the version string was not updated with the backport (as you mentioned, Redhat backports everything under the same version with different release numbers in their RPMs), in any case I see version 18 running on it now:
[ 0.893676] mpt2sas version 18.100.00.00 loaded

FYI - as long as the driver was compiled as a module and not right into the kernel (and its probably only built into the kernel if you compile your own as I do), you can run 'modinfo mpt2sas' to get lots of information about the version installed. Run with 'sudo' if you aren't already root. That command will work whether the driver is loaded or not, so its easy to check the version provided by a distro even if you don't have the hardware (or do it in a VM).

 

[gea]

I have updated the info above with your Solaris recommendation, but what version is that, and do you know when the P19 drivers were added to Solaris/Openindiana?

Since i use and support NexentaCore, OpenSolaris, Illumian, OpenIndiana, Solaris and OmniOS, there are newer mpt_sas drivers now and then but the firmware version was never an item in any discussions so I do not know.

I also do not had any problem, does not matter with an old OpenIndiana and a new LSI adapter nor with a current OmniOS and a quite old HBA.

As I see, this discussion like "use P16" is originally a FreeNAS matter.
Usually you would expect that driver and firmware is up/down compatible
as you cannot expect a dedicated version when you buy a new HBA: It should just work !

If there are problems with some disks, a firmware update/downgrade is usually an
option that may help as a newer driver is not a user option. (P20 is another thing)

- at least under Opensolaris based systems -

 

[Terry Kennedy]

The P20 firmware has been rebuilt at least twice. This LSI knowledgebase article documents 20.00.02.00 from February 2015 (the original 20.00.00.00 was from September 2014). The current download from LSI (for the models I've looked at) is 20.00.04.00 from May 2015.