SAS Expander does not detect more than 8 drives

[butchered-nas]

Hi,

I wanted to extend my (self built) NAS to have more drives than it currently supports. I already had a Fujitsu (LSI branded card) HBA card, that worked well with my drives before the update with the expander. Since the HBA card only has 2 SFF output ports, 8 drives was the limit using SFF to SATA breakout cables.
Now it seems that the expander does not recognize more than 8 drives, even though more are connected to it. It does not matter what drives are connected, when the amount of 8 drives is reached, it stops further drive detection. The drives are not listed per lsscsi.

My NAS is self built and consists of the following components:

CPU: Intel i5 3470
MB: ASUS P8H77-M
RAM: 10 GB DDR3
HBA: Fujitsu D2607-A21 (crossflashed to LSI 9211-8i IT Mode)
SAS Expander: IBM 46M0997 (Firmware: 634A)
PSU: be quiet! Pure Power 550W CM
OS: Ubuntu 22.04.3 LTS

I bought an IBM 46M0097 Expander and it came with firmware version 602A. I updated the firmware (outside of an IBM machine) using the guide from The Art of Server on YouTube and it went well without any error messages. I directly updated from 602A to 634A without any firmware updates in between. I used sg-utils version 1.33.
The HBA is sitting in PCIe 3.0 x16 slot while the expander is sitting in the PCIe 2.0 16x slot. I have a network card sitting the PCIe 3.0 x1 slot.

I also purchased a new SFF-8087 to 4x SATA breakout cable as well as 2 new SFF-8087 to SFF-8087 cables to connect the HBA with the expander. The HBA is connected with both cables with the SAS expander.

The breakout cable seems to work fine since the drives are detected when there are not already 8 drives detected. The expander is not detecting any more drives when I try to connect drives with my other breakout cables that worked fine before the installation.
The two SFF cables also seem to work fine since I tested them individually. To test this, I connected the HBA and expander only with one cable and the 8 drives were detected. However there was also a drive detection limit of 8 drives with one SFF cable each.

There are a couple of questions that I hope you could help me with:
1) Is this more likely a problem with the expander or the HBA card?
2) Since I flashed the expander, can use the same guide I used for the update to flash older firmware version (firmware downgrade)? Or do i need to do additional steps?
3) is it possible that i "bricked" the expander with the firmware update? It is weird since it detects 8 drives without any (visible) error messages.
3) is this more likely a firmware issue or a hardware issue?
3) What more troubleshooting steps could I do to determine the problem?

Thanks in advance.

 

[alaricljs]

More likely to be the hba. I have never seen it on an add-in card, but there are motherboards with onboard hbas that artificially limit the drive count.

 

[Chriggel]

My bet is that this is a problem with the expander. These things can be nasty. The HBA is known to be able to handle more than 8 drives. Way back in the day when this still was a thing for me, I had decent success with expanders from HP and Intel, but not so much with others.

Some expanders don't like the uplink to the HBA being on random ports (although they're like a switch and it shouldn't matter). I see that the IBM expander is already designed to clearly show you which ports are supposed to be the uplink, so I guess that's where you connected the HBA?

 

[butchered-nas]

I see that the IBM expander is already designed to clearly show you which ports are supposed to be the uplink, so I guess that's where you connected the HBA?

Yes, HBA output port labelled as SAS MLC1 is connected to expander port INPUT 1 and HBA output port labelled as SAS MLC2 is connected to expander port INPUT 2.
Breakout cables are connected to expander port BP1, BP2, BP3. BP4 is not used but I also tried it, the expander will stay at 8 detected drives.

I have ordered an IBM M1015 HBA card and will test it with that. When the expander also detects only 8 drives with the IBM HBA, then there is an issue with the expander I guess.

 

[gregsachs]

Dumb idea-have you tried all of the BP ports individually? (IE are you sure you don't have any dead ports on the expander?) Will they all individually see drives?

 

[BLinux]

when you connect both cables from the HBA to the SAS expander, do you see all 8 links between them?

 

[butchered-nas]

Dumb idea-have you tried all of the BP ports individually?

I checked again after your post and it does not detect more than 8 drives, no matter what BP ports are used.

when you connect both cables from the HBA to the SAS expander, do you see all 8 links between them?

How can I check that?

Edit: I think I found what you meant. First I checked the error messages on my Ubuntu machine with
grep -E "(mpt[23]sas| scsi | sd )" /var/log/syslog. I noticed that exceptions were thrown in the mpt part. Then I created a Debian 12 live stick and checked the logs with sudo dmesg | grep -E "(mpt[23]sas| scsi | sd )". Debian used mpt2 instead of mpt3 and no error messages were shown in Debian.

I re-flashed (using the same guide as before) the expander with 634a fimware in Debian in order to rule out that the mpt3 errors in Ubuntu caused any bugs with the expander. However, the expander still can't detect more than 8 drives after the re-flash.
I then found about lsiutil and it seems to me that all links are used.

This is the output of lsiutil commands:

Spoiler: /lsiutil.x86_64 -s

LSI Logic MPT Configuration Utility, Version 1.71, Sep 18, 2013

1 MPT Port found

==============================================================================

ioc0              LSI Logic SAS2008 B2     MPT 200   Firmware 14000700   IOC 0

SAS2008's links are 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G

B___T___L  Type       Vendor   Product          Rev      SASAddress     PhyNum
0   0   0  Disk       ATA      TOSHIBA MG07ACA1 0104  5005076028d8b2ed    12
0   1   0  Disk       ATA      WDC WD100EMAZ-00 0A83  5005076028d8b2ee    13
0   2   0  Disk       ATA      WDC WD101EDBZ-11 0A85  5005076028d8b2ef    14
0   3   0  Disk       ATA      WDC WD101EDBZ-11 0A85  5005076028d8b2f0    15
0   4   0  Disk       ATA      HGST HDN724040AL A5E0  5005076028d8b2f1    16
0   5   0  Disk       ATA      ST5000DM000-1FK1 CC49  5005076028d8b2f2    17
0   6   0  Disk       ATA      ST5000DM000-1FK1 CC49  5005076028d8b2f3    18
0   7   0  Disk       ATA      ST6000DM003-2CY1 0001  5005076028d8b2f4    19
0   8   0  EnclServ   IBM-ESXS SAS EXP BP       634A  5005076028d8b307    38

Spoiler: /lsiutil.x86_64 -p 1 -e -> Menu 68 (Show port state)

Current Port State
------------------
SAS2008's links are 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G

Software Version Information
----------------------------
Current active firmware version is 14000700 (20.00.07)
Firmware image's version is MPTFW-20.00.07.00-IT
  LSI Logic
  Not Packaged Yet
x86 BIOS image's version is MPT2BIOS-7.39.02.00 (2015.08.03)

Firmware Settings
-----------------
SAS WWID:                       500300570XXXXXX
Multi-pathing:                  Disabled
SATA Native Command Queuing:    Enabled
--more, hit RETURN--          
SATA Write Caching:             Enabled
SATA Maximum Queue Depth:       32
SAS Max Queue Depth, Narrow:    0
SAS Max Queue Depth, Wide:      0
Device Missing Report Delay:    0 seconds
Device Missing I/O Delay:       0 seconds
Phy Parameters for Phynum:      0    1    2    3    4    5    6    7  
  Link Enabled:                 Yes  Yes  Yes  Yes  Yes  Yes  Yes  Yes
  Link Min Rate:                1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5
  Link Max Rate:                6.0  6.0  6.0  6.0  6.0  6.0  6.0  6.0
  SSP Initiator Enabled:        Yes  Yes  Yes  Yes  Yes  Yes  Yes  Yes
  SSP Target Enabled:           No   No   No   No   No   No   No   No 
  Port Configuration:           Auto Auto Auto Auto Auto Auto Auto Auto
Interrupt Coalescing:           Enabled, timeout is 10 us, depth is 4

Spoiler: /lsiutil.x86_64 -p 1 -e -> Menu 16 (Show attached devies)

SAS2008's links are 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G

B___T     SASAddress     PhyNum  Handle  Parent  Type
        50030057012f6140           0001           SAS Initiator
        50030057012f6141           0002           SAS Initiator
        50030057012f6142           0003           SAS Initiator
        50030057012f6143           0004           SAS Initiator
        50030057012f6144           0005           SAS Initiator
        50030057012f6145           0006           SAS Initiator
        50030057012f6146           0007           SAS Initiator
        50030057012f6147           0008           SAS Initiator
        5005076028d8b2e0     0     0009    0001   Edge Expander
0   0  5005076028d8b2ed    12     000a    0009   SATA Target
0   8  5005076028d8b307    38     000b    0009   SAS Initiator and Target
0   4  5005076028d8b2f1    16     000c    0009   SATA Target
0   5  5005076028d8b2f2    17     000d    0009   SATA Target
0   1  5005076028d8b2ee    13     000e    0009   SATA Target
0   6  5005076028d8b2f3    18     000f    0009   SATA Target
0   7  5005076028d8b2f4    19     0010    0009   SATA Target
0   2  5005076028d8b2ef    14     0011    0009   SATA Target
0   3  5005076028d8b2f0    15     0012    0009   SATA Target

Type      NumPhys    PhyNum  Handle     PhyNum  Handle  Port  Speed
Adapter      8          0     0001  -->    3     0009     0    6.0
                        1     0001  -->    2     0009     0    6.0
                        2     0001  -->    1     0009     0    6.0
                        3     0001  -->    0     0009     0    6.0
                        4     0001  -->    7     0009     0    6.0
                        5     0001  -->    6     0009     0    6.0
                        6     0001  -->    5     0009     0    6.0
                        7     0001  -->    4     0009     0    6.0

Expander    39          0     0009  -->    3     0001     0    6.0
                        1     0009  -->    2     0001     0    6.0
                        2     0009  -->    1     0001     0    6.0
                        3     0009  -->    0     0001     0    6.0
                        4     0009  -->    7     0001     0    6.0
                        5     0009  -->    6     0001     0    6.0
                        6     0009  -->    5     0001     0    6.0
                        7     0009  -->    4     0001     0    6.0
                       12     0009  -->    0     000a     0    6.0
                       13     0009  -->    0     000e     0    6.0
                       14     0009  -->    0     0011     0    6.0
                       15     0009  -->    0     0012     0    6.0
                       16     0009  -->    0     000c     0    6.0
                       17     0009  -->    0     000d     0    6.0
                       18     0009  -->    0     000f     0    6.0
                       19     0009  -->    0     0010     0    6.0
                       38     0009  -->    0     000b     0    6.0

Enclosure Handle   Slots       SASAddress       B___T (SEP)
           0001      8      50030057012f6140
           0002     39      5005076028d8b2e0    0   8

The other drives are also connected to the power supply and they are also detected by the expander when I unplug a SATA cable from a dectected drives and plug it onto an undetected drive.

 

[BLinux]

@butchered-nas ok, the 8 SAS links look good. I think something you can try is to reset the configuration on the HBA controller. also, what is the PCIe class / vendor / product ID codes? can you show a lspci of the card?

I show how to factory reset LSI card firmware configuration in this video at this point:

 

[butchered-nas]

Actually I found the issue. SAS Expander and both HBA work fine.
Most of my drives are shucked external WD drives that don't power on when a certain pin on SATA power connector provides 3.3V.
I re-arranged my drives in my case when I installed the expander. Two of my three SATApower cables are modified to exlude 3.3V, but I attached the unmodified cable (where 3.3v is present) to my shucked drives and thus they werent spinning up. This is why only 8 drives were detected, because the other 4 wouldnt power on.

 

[Sean Ho]

SATA PWDIS strikes again!