Misadventures with LSI SAS3008 cards

[Spearfoot]

This Spring I bought 3 of Supermicro's AOC-S3008L-L8e cards, with the idea of upgrading the 3 LSI SAS 9210-8i HBAs in my main X9DRi-LN4F+-based FreeNAS server. These should just be a 'drop-in' upgrade, right?

Wrong! I got all kinds of errors, like this:

(da2:mpr0:0:0:0): READ(10). CDB: 28 00 e8 e0 87 80 00 01 00 00 
(da2:mpr0:0:0:0): CAM status: SCSI Status Error
(da2:mpr0:0:0:0): SCSI status: Check Condition
(da2:mpr0:0:0:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
(da2:mpr0:0:0:0): Retrying command (per sense data)

Well... since I am passing through the 3 HBAs to a FreeNAS VM running under ESXi 6.7 -- hey, I know! but this is my home lab! and it's worked flawlessly for years! -- I thought that might be the problem. So I moved my testing over to another system (an X9SRL-F) where I could run both ESXi and bare metal.

Same results either way. So it's not an artifact of passing the cards through to a VM.

The cards came with phase 16.00.01.00 firmware, so I then tried Supermicro's phase 16.00.10.00, and then Broadcom's 16.00.10.00, and then Broadcom's pre-release 16.00.12.00 (courtesy of iXsystems)... all with the same results.

I found this scary thread at bugs.freebsd.org, in which quite a few users report similar problems with LSI HBAs (don't be fooled by the title; the problem isn't just with Seagate drives):

224496 – mpr and mps drivers seems to have issues with large seagate drives

bugs.freebsd.org

I tried a few BIOS tricks:

• Increased the PCI latency from 64 to 128
• Changed PCIe Maximum Payload from 'Auto' to the largest value
• Changed PCIe Maximum Read Request from 'Auto' to the largest value
• Disabled Power Technology

But none of these made any difference either.

I'm using good Supermicro cables, and I have six of them -- so I doubt that's the problem.

Have I simply had the bad luck to stumble across the FreeBSD driver bug? Are SAS3008 cards just incompatible with my X9-series Supermicro systems? It doesn't seem likely that I bought three lemons, but I guess that's possible, too.

Thanks in advance for your thoughts, my friends!

 

[absth]

First of all I'd check if there's enough cooling for the cards.
Do all ports experience problems?
To exclude FreeBSD driver problems I'd boot up recent enough Linux from live CD/USB and put some load on disks with it.

 

[Spearfoot]

Thanks guys!

First of all I'd check if there's enough cooling for the cards.

Yes, those rascals do run hot! On the work bench I always point a fan at 'em. In the chassis I mount a fan directly over them, like this:
]

Do all ports experience problems?

It seems to be random.

To exclude FreeBSD driver problems I'd boot up recent enough Linux from live CD/USB and put some load on disks with it.

Good idea! I'll try that out when I get a chance.

I agree with absth. I also think that the MPR driver isnt compatible with IT Mode firmware, so you could try to make Freenas/Truenas load the MPS driver. I know some people were getting problems with this before.

Thank you, sir! That's a good hint. FreeNAS/TrueNAS do pick the MPS driver for my LSI9210 cards and the SAS2308 on my X10SL7-F system. Not sure why they pick the MPR driver for the SAS3008 chip.

Funny thing is, I have a pair of X11DPH-T servers at work with the same HBA, running FreeNAS 11.2U8 with the MPR driver -- and they work fine! Maybe I just need to get Supermicro to wave their magic wand over my systems? Ha!

 

[T_Minus]

My 02.... That fan is not enough for those 3 cards stacked so close.

 

[ericloewe]

FreeNAS/TrueNAS do pick the MPS driver for my LSI9210 cards and the SAS2308 on my X10SL7-F system. Not sure why they pick the MPR driver for the SAS3008 chip.

That's exactly as it should be. The SAS1 driver is mpt, SAS2 is mps, SAS3 is mpr. I'm not sure if mpr also covers the SAS4 stuff, but I think it does cover the SAS3.5 stuff.

 

[Spearfoot]

My 02.... That fan is not enough for those 3 cards stacked so close.

Could be... but I don't think so.

There are 5 chassis fans in the CSE-846 case, plus active coolers running on both CPUs, plus the fan in the slot area -- sited in my climate-controlled shop. I run a fan script on the server to set the fan mode depending on CPU and HDD temperatures. Most of the time it runs at the 'Standard' setting, with the rear chassis fans spinning at 3000RPM and the fronts at 3500. Got pretty good air flow in this guy...

 

[Spearfoot]

That's correct. The mpr driver has a known bug with Freenas with sas3008 chipsets and others. I'm not sure if this was ever fixed.

This is what I'm afraid of... if you read through the FreeBSD MPR and MPS thread I linked above, you'll find people having the same issues with both drivers -- it's not just the SAS3008 HBAs that have problems.

My 3 x 9210's have run flawlessly for 4 years, since I built this server. But both FreeNAS 11.2U8 and TrueNAS 12.0U5 -- which I believe are based on FreeBSD 11.2 and 12.0 respectively -- barf as soon as I drop in the SAS3008 cards.

Baffling!

 

[T_Minus]

Could be... but I don't think so.

There are 5 chassis fans in the CSE-846 case, plus active coolers running on both CPUs, plus the fan in the slot area -- sited in my climate-controlled shop. I run a fan script on the server to set the fan mode depending on CPU and HDD temperatures. Most of the time it runs at the 'Standard' setting, with the rear chassis fans spinning at 3000RPM and the fronts at 3500. Got pretty good air flow in this guy...

With the chassis fans I wouldn't run the fan above it then like that, I thought that was your only cooling, I missed it or misread

 

[Spearfoot]

Okay, I believe I can definitely claim to have a bad batch of Supermicro AOC-3008L-L8e boards.

I've tested the 3 cards in 3 different systems, against a known-good ZFS pool of 8 x 2TB HDDs.

Flashed 'em six ways from Sunday (DOS sas3flsh, UEFI sas3flash.efi, Linux & FreeBSD sas3flash) with four versions of IT mode firmware (SMC 16.00.10.00 & 16.00.01.00, LSI 16.00.12.00 & 16.00.10.00). Still no joy.

Used DOS megarec3 to write new SBRs and erase the flash, without success. Also tried out lsiutil v1.72, lsirec, and sbrtool.py to try and coax these cards into working. No luck.

In the meantime, I bought an LSI SAS9300-8i, which worked great right off the bat. So it's not like my test systems can't work with the SAS3008 chip.

It's long past the 30-day return window, but I've contacted the eBay seller to see what they have to say.

On a side note, this is the same eBay seller I bought a Supermicro X9SRL-F board from recently, which I had to return -- the BMC died. I'm batting zero with them so far -- 4 items purchased, and all of them failed.

Them's the breaks, I guess!

 

[Spearfoot]

Just heard back from the eBay seller, basically telling me 'tough luck'.

Ah, well. Caveat Emptor!