Just finished setting up a new all flash raid array in a supermicro CSE-836 case with BPN-SAS2-836EL2 backplane and are having some weird issue. The SSD i am using is Lenovo OEM HUSSM1680 800gb variance.
The logs are created whenever there's a heavy load on the array and I tried upgrading both the controller firmware to the latest P16, as well as replace the cable. I also tried shuffling the ssd just so i eliminate the issue with particular drive. Unfortunately the issue persist although much less that it was before I replace the cable & upgrade the firmware.
Here are the log I am seeing:
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 30 98 38 c8 00 00 20 00
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): Retrying command (per sense data)
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 30 6f 2b 78 00 00 10 00
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): Retrying command (per sense data)
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 23 fc a3 e0 00 00 08 00
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
The server is running the latest stable freenas (FreeNAS-11-STABLE) and it is setup using active/active multipath.
When the issue occuring, there's a major drop in performance which you can see during activities such as svmotion between 2 different iscsi zvol.
One thing I have not tried is to replace the controller with the onboard SAS2308 controller, this way both controller and backplane are all 6gb, and I have supermicro sas cable just to eliminate the possibility of bad generic quality cable.
Anyone have seen similar issue ?? any idea or suggestion what I should try next to troubleshoot the issue ?
The logs are created whenever there's a heavy load on the array and I tried upgrading both the controller firmware to the latest P16, as well as replace the cable. I also tried shuffling the ssd just so i eliminate the issue with particular drive. Unfortunately the issue persist although much less that it was before I replace the cable & upgrade the firmware.
Here are the log I am seeing:
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 30 98 38 c8 00 00 20 00
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
Nov 21 02:06:46 freenas (da9:mpr0:0:27:0): Retrying command (per sense data)
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 30 6f 2b 78 00 00 10 00
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
Nov 21 02:07:06 freenas (da9:mpr0:0:27:0): Retrying command (per sense data)
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): READ(10). CDB: 28 00 23 fc a3 e0 00 00 08 00
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): CAM status: SCSI Status Error
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): SCSI status: Check Condition
Nov 21 02:07:53 freenas (da9:mpr0:0:27:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
The server is running the latest stable freenas (FreeNAS-11-STABLE) and it is setup using active/active multipath.
When the issue occuring, there's a major drop in performance which you can see during activities such as svmotion between 2 different iscsi zvol.
One thing I have not tried is to replace the controller with the onboard SAS2308 controller, this way both controller and backplane are all 6gb, and I have supermicro sas cable just to eliminate the possibility of bad generic quality cable.
Anyone have seen similar issue ?? any idea or suggestion what I should try next to troubleshoot the issue ?