LSI 9300-8i Freenas SCSI Error Logs

[azev]

Just finished setting up a new all flash raid array in a supermicro CSE-836 case with BPN-SAS2-836EL2 backplane and are having some weird issue. The SSD i am using is Lenovo OEM HUSSM1680 800gb variance.
The logs are created whenever there's a heavy load on the array and I tried upgrading both the controller firmware to the latest P16, as well as replace the cable. I also tried shuffling the ssd just so i eliminate the issue with particular drive. Unfortunately the issue persist although much less that it was before I replace the cable & upgrade the firmware.
Here are the log I am seeing:

Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 30 98 38 c8 00 00 20 00
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): Retrying command (per sense data)
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 30 6f 2b 78 00 00 10 00
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): Retrying command (per sense data)
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 23 fc a3 e0 00 00 08 00
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)

The server is running the latest stable freenas (FreeNAS-11-STABLE) and it is setup using active/active multipath.

When the issue occuring, there's a major drop in performance which you can see during activities such as svmotion between 2 different iscsi zvol.

One thing I have not tried is to replace the controller with the onboard SAS2308 controller, this way both controller and backplane are all 6gb, and I have supermicro sas cable just to eliminate the possibility of bad generic quality cable.

Anyone have seen similar issue ?? any idea or suggestion what I should try next to troubleshoot the issue ?

 

[Rand__]

Cable or drive I'd say

 

[PnoT]

I second a fresh cable being put in and test.

 

[Dave Corder]

I vaguely recall hearing about something similar with a 92xx card that turned out to be inadequate cooling of the controller. How's your cooling setup?

 

[azev]

Cable or drive I'd say

I second a fresh cable being put in and test.

Ok, I plan to buy a supermicro cable from supermicro store and see if that helps.

I vaguely recall hearing about something similar with a 92xx card that turned out to be inadequate cooling of the controller. How's your cooling setup?

Hmmm I can't see the temp of the controller via cli in freenas or at least I dont know how... But I assume supermicro 836 chassis have sufficient cooling from the factory. I even set the fan speed at maximum just to see if it was cooling, but it still throwing the same error.

I regret swapping cables and updating firmware the same time, now I am not sure what actually helps during the initial troubleshooting.

Maybe I'll just drop the 9300-8i controller and just use the onboard 2308 controller, the reason I use 9300-8i is to eliminate bottleneck, but if I think about it again, the backplane is the bottleneck since this is all SSD setup.