Issues with LSI2008 on OmniOS? (mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00)

I'm getting loads of issue on my napp-it(omniOS) box, all relating to my m1015 / onboard LSI2008.

The log is kind of flooded with entried like below, and I get transport errors in iostat.

It's conserning all 2008 cards I have, ever tried another one, but am getting errors still. It's basically different drives taht are showing for errors too...

I would really apperciate any help to point me in a direction where to look? This is not a lot of information, but I will post anything that is needed... (Could be looking into getting another HBA, if needed.)

Oct 30 16:39:10 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:39:10 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:39:10 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:39:10 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:39:11 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:39:11 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:39:11 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:39:11 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:40:01 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:40:01 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:40:01 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:40:01 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:40:01 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:40:01 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:03 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:40:03 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:03 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:40:03 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:05 napp-it-14b scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:40:05 napp-it-14b Log info 0x31110d00 received for target 13.
Oct 30 16:40:05 napp-it-14b scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc
Oct 30 16:40:08 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:40:08 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:08 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:40:08 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:40:12 napp-it-14b scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:40:12 napp-it-14b Log info 0x31110d00 received for target 13.
Oct 30 16:40:12 napp-it-14b scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc
Oct 30 16:41:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:41:01 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:41:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:41:01 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:41:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:41:01 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:41:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:41:01 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:42:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:42:01 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:42:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:42:01 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:42:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:42:01 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:42:01 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:42:01 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:43:02 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:43:02 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:43:02 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:43:02 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:43:02 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:43:02 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:43:02 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Oct 30 16:43:02 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:43:02 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:43:02 napp-it-14b mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d00
Oct 30 16:43:02 napp-it-14b scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@18/pci1000,3020@0 (mpt_sas20):
Oct 30 16:43:02 napp-it-14b mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d00

 

This will lead you down a rabbit hole that may very well be useful: illumos gate - Bug #4189: sas multipath - illumos.org
Could you give a bit more information about your storage topology?

 

what I would try
- disable mpio (menu disk - details - edit mpt-sas), followed by a reboot
- what LSI firmware are you using - there are reports about problems on P20 (use P19)
- update to newest OmniOS 151012

 

Woops, sorry for not updating - I found what MY issue was at least...and, I am "sad" to say it was something as trivial and easy as not enough power to the drives! Someone in a IRC channel pointed out that there were some reset logs and I started listening to my drives - HEUREKA I swapped out my molex connectors and made sure each vdev had a dedicated line of connectors to the PSU.

After that all was fine... I pulled my hair over such an easy fix. I do not know how it came about, since I hadn't changed anything from the start, but maybe it was just on the "edge" and my PSU got degraded or something.

Not sure how to update or inform the people in the mailing list, but maybe they do have the same issue...

 

Looks like it happened to me at well. Mdadm would lose 3-4 hdds at once - it totally looked like a faulty port on a controller. At first I was seeing lots of 0x31111000, but today I finally got 0x31110d00 and was able to find this solution. Turned out that I had a molex splitter on a bottommost backplane (norco 4224 clone, 6 little backplanes each powered by a molex) and that's where drives were giving me a headache.

Removed the spiltter and it seems to work now.