troubleshoot SAS3 expander

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Lleo_

New Member
Dec 27, 2014
9
2
3
34
Hi,
I built a new storage system using a Norco DS-24D, to which I added an Intel RES3TV360 SAS3 expander. In the enclosure I used LSI CBL-SFF8643-8087 mini-SAS_to_mini-SAS-HD cables.

The expander is also connected to a SAS-AD4443-2 8-port SFF-8644 to SFF-8643 12G HD Mini SAS Adapter.

The enclosure is connected using SFF-8644 to SFF-8644 External SAS3 Cable to Intel RS3GC008 HBA in my server.

I added 12x4TB WD Red drives and 6 additional miscellaneous SATA drives that I had laying here.

My (test) server is running Ubuntu, no ESXi or virtualization is involved.
All drives are visible to the OS, but I am getting confusing output from the sas3ircu utility.
The utility shows as I have two enclosures attached.

Code:
LSI Corporation SAS3 IR Configuration Utility.
Version 07.00.00.00 (2014.08.14)
Copyright (c) 2009-2014 LSI Corporation. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.13.00.00
  Firmware version                        : 6.00.00.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 1023
  Concurrent commands supported           : 10176
  Slot                                    : 7
  Segment                                 : 0
  Bus                                     : 1
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #  : 0
  Slot #  : 0
  SAS Address  : 50000d1-1-087f-ae98
  State  : Ready (RDY)
  Size (in MB)/(in sectors)  : 3815447/7814037167
  Manufacturer  : ATA   
  Model Number  : WDC WD40EFRX-68W
  Firmware Revision  : 0A82
  Serial No  : WDWCC4E0TA83JY
  GUID  : 50014ee2606667f3
  Protocol  : SATA
  Drive Type  : SATA_HDD

...snip...
Code:
...snip...
Device is a Hard disk
  Enclosure #  : 2
  Slot #  : 19
  SAS Address  : 50000d1-1-087f-ae93
  State  : Ready (RDY)
  Size (in MB)/(in sectors)  : 953869/1953525167
  Manufacturer  : ATA   
  Model Number  : SAMSUNG HD103UI
  Firmware Revision  : 1111
  Serial No  : S1MNJ1NQA33359
  GUID  : 50000f00003a3395
  Protocol  : SATA
  Drive Type  : SATA_HDD

Device is a Enclosure services device
  Enclosure #  : 2
  Slot #  : 27
  SAS Address  : 50000d1-1-087f-aebe
  State  : Standby (SBY)
  Manufacturer  : INTEL   
  Model Number  : RES3TV360   
  Firmware Revision  : B012
  Serial No  : PMCSIERAD
  GUID  : N/A
  Protocol  : SAS
  Device Type  : Enclosure services device
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#  : 1
  Logical ID  : 500605b0:071397e0
  Numslots  : 8
  StartSlot  : 0
  Enclosure#  : 2
  Logical ID  : 50000d11:087faebe
  Numslots  : 28
  StartSlot  : 0
------------------------------------------------------------------------
SAS3IRCU: Command DISPLAY Completed Successfully.
SAS3IRCU: Utility Completed Successfully.
I could not post the entire output from the utility, but some drives are shown to have no enclosure and slot info and other do. This is regardless how the cables, drives are plugged in.
This is the output filtered for slot #s:
Slot # : 0
Slot # : 0
Slot # : 0
Slot # : 0
Slot # : 0
Slot # : 0
Slot # : 0
Slot # : 0
Slot # : 4
Slot # : 5
Slot # : 6
Slot # : 7
Slot # : 12
Slot # : 13
Slot # : 14
Slot # : 15
Slot # : 18
Slot # : 19
Slot # : 27

Could the first two backplanes be bad?
Any idea how to troubleshoot this further?
Thanks a lot in advance!
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
I can't explain why it is doing it, but would guess that it is listing the HBA as the first enclosure, and the expander as the second enclosure. And so assuming that you are not connecting any drives directly to the HBA all of the drives should show up belonging to enclosure 2.

It may or may not help, but I'm curious as to what the OS thinks is going on, instead of LSI's CLI. Note that you may need to 'apt-get install' a couple things for these commands to work. First, use 'lsscsi -g' to find the /dev/sg## device name for your expander. Then, run 'sg_ses -j /dev/sg##' (using the device from the previous command). This will effectively be using generic tools to ask the expander what it thinks about everything and keeping the HBA out of the picture except to transport the SAS commands/data. And FYI you can also use 'sg_ses -j -f /dev/sg##' to filter the data down (or -ff for more filtering).
 
  • Like
Reactions: Lleo_

Lleo_

New Member
Dec 27, 2014
9
2
3
34
Thanks, this was already really helpful.
So outputs are here:

Code:
root@archive:~# lsscsi -g
[2:0:0:0]    disk    ATA      INTEL SSDSA2M080 2CV1  /dev/sda   /dev/sg0
[8:0:0:0]    disk    ATA      SAMSUNG HD103UI  1113  /dev/sdb   /dev/sg1
[8:0:1:0]    disk    ATA      SAMSUNG HD103UI  1111  /dev/sdc   /dev/sg2
[8:0:2:0]    disk    ATA      SAMSUNG HD103UI  1111  /dev/sdd   /dev/sg3
[8:0:3:0]    disk    ATA      SAMSUNG HD103UI  1113  /dev/sde   /dev/sg4
[8:0:4:0]    disk    ATA      WDC WD40EFRX-68W 0A80  /dev/sdf   /dev/sg5
[8:0:5:0]    disk    ATA      WDC WD40EFRX-68W 0A80  /dev/sdg   /dev/sg6
[8:0:6:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdh   /dev/sg7
[8:0:7:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdi   /dev/sg8
[8:0:8:0]    disk    ATA      WDC WD20EARS-00M AB51  /dev/sdj   /dev/sg9
[8:0:9:0]    disk    ATA      SAMSUNG HD103UI  1111  /dev/sdk   /dev/sg10
[8:0:10:0]   disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdl   /dev/sg11
[8:0:11:0]   disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdm   /dev/sg12
[8:0:12:0]   disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdn   /dev/sg13
[8:0:13:0]   disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdo   /dev/sg14
[8:0:14:0]   disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdp   /dev/sg15
[8:0:15:0]   disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdq   /dev/sg16
[8:0:16:0]   disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdr   /dev/sg17
[8:0:17:0]   disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sds   /dev/sg18
[8:0:18:0]   enclosu INTEL    RES3TV360        B012  -          /dev/sg19
and the enclosure reports this:



This seems all OK, the slots and their status seems accurate, should I just ignore sas3ircu?
 

Attachments

Lleo_

New Member
Dec 27, 2014
9
2
3
34
So not sure what changed but after a cold restart it seems that I figured it out.
Also read again the hardware guide for the expander. http://download.intel.com/support/motherboards/server/res3tv360/sb/res3tv360_ug.pdf
On page #9 it is mentioned as standard expander feature that there are 8 inputs and 28 outputs. This seemed familiar as those were the slots numbers showed for the different phantom enclosures.
By choice I always connected the last ports on the expander to the HBA, but this time around I put them first, using the SAS connector CN0 and CN1 on the expander. Than the remainder of the back-planes, skipping connector 4 (CN4) see page #15.
Now all shows correctly both on sas3ircu and lsscsi commands.
Thanks for all your help.
 
Last edited: