LSI 3108 on X10DRH-CT not able to use all my drives?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
L

lastb0isct

New Member
Feb 12, 2018
24
2
3
38
Quick rundown:

RHEL8 running OpenZFS with 35 drives passed to it. I was previously accomplishing this directly with a SAS2008 card because my previous chassis was a SAS-2 backplane which didn't seem to play nice with the 3108. The current state of the system is when I plug-in both backplanes (front = 24 drives | back = 12 drives) the 3108 is able to see all of the drives but within the BIOS it shows them as "Blocked" & I can't take any action on them. From the OS via storcli it shows them as Unconfigured Bad Unsupported drives (previously they were Unconfigured Good Unsupported but I have attempted to correct it to no avail).

Here are the outputs I get when I am connecting all of the drives to the 3108 -- you can see that the set good commands fail for both:


Code: 
PD LIST :
=======

-----------------------------------------------------------------------------------
EID:Slt DID State  DG      Size Intf Med SED PI SeSz Model                 Sp Type
-----------------------------------------------------------------------------------
40:1     47 JBOD   -  12.733 TB SATA HDD N   N  512B OOS14000G             U  -
40:2     48 JBOD   -  12.733 TB SATA HDD N   N  512B OOS14000G             U  -
40:3     49 JBOD   -  12.733 TB SATA HDD N   N  512B OOS14000G             U  -
40:5     41 JBOD   -  10.914 TB SATA HDD N   N  512B WDC WD120EMFZ-11A6JA0 U  -
40:6     42 JBOD   -  10.914 TB SATA HDD N   N  512B WDC WD120EMFZ-11A6JA0 U  -
40:7     46 JBOD   -  14.552 TB SATA HDD N   N  512B WDC WD161KRYZ-01AGBB0 U  -
40:8     45 JBOD   -  14.552 TB SATA HDD N   N  512B WDC WD161KRYZ-01AGBB0 U  -
40:9     44 JBOD   -  12.733 TB SATA HDD Y   N  512B WDC WD140EDGZ-11B1PA0 U  -
40:10    43 JBOD   -  12.733 TB SATA HDD Y   N  512B WDC WD140EDGZ-11B1PA0 U  -
40:11    50 JBOD   -  16.371 TB SATA HDD N   N  512B ST18000NM000J-2TV103  U  -
40:12    51 JBOD   -  16.371 TB SATA HDD N   N  512B ST18000NM000J-2TV103  U  -
52:1     56 JBOD   -   7.277 TB SATA HDD Y   N  512B HUH728080ALE601       U  -
52:2     66 UBUnsp -   7.276 TB SATA HDD N   N  512B WDC WD80EMAZ-00W      U  -
52:3     53 JBOD   -   7.277 TB SATA HDD N   N  512B WDC WD80EMAZ-00WJTA0  U  -
52:4     62 UBUnsp -   7.276 TB SATA HDD Y   N  512B WDC WD80EZAZ-11T      U  -
52:5     54 JBOD   -   7.277 TB SATA HDD N   N  512B WDC WD80EMAZ-00WJTA0  U  -
52:6     60 UBUnsp -   7.276 TB SATA HDD N   N  512B WDC WD80EMAZ-00M      U  -
52:7     59 UBUnsp -   7.276 TB SATA HDD N   N  512B WDC WD80EMAZ-00W      U  -
52:8     64 UBUnsp -   7.276 TB SATA HDD Y   N  512B WDC WD80EDAZ-11T      U  -
52:9     58 UBUnsp -   7.276 TB SATA HDD N   N  512B WDC WD80EMAZ-00M      U  -
52:10    55 JBOD   -   7.277 TB SATA HDD N   N  512B WDC WD80EMAZ-00WJTA0  U  -
52:11    76 UBUnsp -   7.276 TB SATA HDD N   N  512B OOS8000G              U  -
52:12    68 UBUnsp -   7.276 TB SATA HDD Y   N  512B WDC WD80EDAZ-11T      U  -
52:13    57 JBOD   -   7.277 TB SATA HDD N   N  512B WDC WD80EMAZ-00WJTA0  U  -
52:14    61 UBUnsp -   7.276 TB SATA HDD N   N  512B WDC WD80EMAZ-00W      U  -
52:15    67 UBUnsp -   7.276 TB SATA HDD N   N  512B WDC WD80EMAZ-00W      U  -
52:16    69 UBUnsp -   7.276 TB SATA HDD N   N  512B WDC WD80EMAZ-00W      U  -
52:17    63 UBUnsp -   7.276 TB SATA HDD N   N  512B WDC WD80EMAZ-00M      U  -
52:18    65 UBUnsp -   7.276 TB SATA HDD N   N  512B HGST HUS728T8TAL      U  -
52:19    70 UBUnsp -  12.732 TB SATA HDD N   N  512B OOS14000G             U  -
52:20    71 UBUnsp -  12.732 TB SATA HDD N   N  512B OOS14000G             U  -
52:21    74 UBUnsp -  12.732 TB SATA HDD N   N  512B OOS14000G             U  -
52:22    73 UBUnsp -  12.732 TB SATA HDD N   N  512B OOS14000G             U  -
52:23    75 UBUnsp -  12.732 TB SATA HDD N   N  512B OOS14000G             U  -
52:24    72 UBUnsp -  12.732 TB SATA HDD N   N  512B OOS14000G             U  -
-----------------------------------------------------------------------------------




./storcli64 /c0/e52/s24 set good                                                                                                                                                    PLEX
CLI Version = 007.1704.0000.0000 Jan 16, 2021
Operating system = Linux 4.18.0-425.10.1.el8_7.x86_64
Controller = 0
Status = Failure
Description = Set Drive Good Failed.

Detailed Status :
===============

-------------------------------------------------
Drive       Status  ErrCd ErrMsg
-------------------------------------------------
/c0/e52/s24 Failure   255 Operation not allowed.
-------------------------------------------------


MegaCLI:

megacli -PDMakeGood '-PhysDrv[52:24]' -a0                                                                                                                              PLEX

Adapter: 0: Failed to change PD state at EnclId-52 SlotId-24.

Exit Code: 0x01
Does anyone have any suggestions on how to fix this?

Currently I am in a working state by plugging in the front backplane (24 drives) into the LSI 2008 & the rear backplane (12 drives) into the 3108 and that seems to be working fine but it is less than ideal. I'd love to have it all just plugged into the 3108 and get rid of the 2008.
 
Last edited:
S

Stankyjawnz

Member
Aug 2, 2017
50
13
8
35
Did you try using the bios on the controller to enable the drives? Should be able to access with ctrl h on boot.
 
L

lastb0isct

New Member
Feb 12, 2018
24
2
3
38
It's Ctrl+r to access its megaraid utility and it doesn't give any options in the bios to unblock them
 
L

lastb0isct

New Member
Feb 12, 2018
24
2
3
38
Here is the output from storcli on the drive now:


Code: 
CLI Version = 007.1704.0000.0000 Jan 16, 2021
Operating system = Linux 4.18.0-425.10.1.el8_7.x86_64
Controller = 0
Status = Failure
Description = Show Drive Information Failed.

Detailed Status :
===============

---------------------------------
Drive       Status  ErrCd ErrMsg
---------------------------------
/c0/e52/s24 Failure    46 -
---------------------------------



Drive Information :
=================

------------------------------------------------------------------------------
EID:Slt DID State  DG      Size Intf Med SED PI SeSz Model            Sp Type
------------------------------------------------------------------------------
52:24    72 UGUnsp -  12.732 TB SATA HDD N   N  512B OOS14000G        U  -
------------------------------------------------------------------------------

EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild
 
L

lastb0isct

New Member
Feb 12, 2018
24
2
3
38
Additional information -- storcli shows that the controller doesn't support JBOD, even though it does!

storcli64 /c0 show jbod
CLI Version = 007.1704.0000.0000 Jan 16, 2021
Operating system = Linux 4.18.0-425.10.1.el8_7.x86_64
Controller = 0
Status = Failure
Description = None

Detailed Status :
===============

-------------------------------------------
Ctrl_Prop Value
-------------------------------------------
JBOD Controller does not support JBOD
-------------------------------------------
 
A

alaricljs

Active Member
Jun 16, 2023
199
74
28
storcli doesn't even show a controller on my X10DRH-CT... which is set to JBOD per the BIOS:
23-window.jpg

I can jam my collection of pointless sata drives to try the >16 question later.
 
L

lastb0isct

New Member
Feb 12, 2018
24
2
3
38
Hmmm, interesting. If you have more than 16 drives none of the drives populate in the OS (lsblk/blkid) but storcli/megacli always show up the controller at least.

This doesn't show anything?

Code: 
storcli /c0 show all
 
A

alaricljs

Active Member
Jun 16, 2023
199
74
28
# storcli show
CLI Version = 007.2405.0000.0000 Sep 28, 2022
Operating system = Linux 6.3.9-arch1-1
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = redacted
Operating System = Linux 6.3.9-arch1-1
 
L

lastb0isct

New Member
Feb 12, 2018
24
2
3
38
You're sure you have a X10DRH-CT? Strange that you don't have it show up there. Also my BIOS I was looking in the MegaRAID Config Tool which is ctrl+r after initial boot screen
 
A

alaricljs

Active Member
Jun 16, 2023
199
74
28
That looks like BIOS behavior whereas I'm running UEFI.

# storcli show && dmidecode |grep X10
CLI Version = 007.2405.0000.0000 Sep 28, 2022
Operating system = Linux 6.3.9-arch1-1
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = redacted
Operating System = Linux 6.3.9-arch1-1


Product Name: X10DRH
Family: SMC X10
Product Name: X10DRH-CT
String 2: Supermicro Motherboard-X10 Series
 
L

lastb0isct

New Member
Feb 12, 2018
24
2
3
38
I'm running UEFI as well but might be running both still. Does this show a controller there? Might need drivers or update the firmware on the card?

Code: 
lspci | grep LSI
 
A

alaricljs

Active Member
Jun 16, 2023
199
74
28
It's running all my storage aside from OS, so it's definitely there.
# lspci |grep LSI
01:00.0 Serial Attached SCSI controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

I also just upgraded to FW 4.680.00-8561 on it, I see no difference tho.
 
M

MoMeanMugs

Member
Apr 16, 2018
60
19
8
74
This is a vendor limitation imposed by SuperMicro. The crappy part is this was never listed on their product pages until well after the products were sold and in the wild. They've since updated their product pages to list the 16 drive limit. I've mentioned this in other threads.

OP, where are you located? I am willing to help, but I am about to leave town and won't be back for a week (need to travel the following week as well). You can send me a PM.
 
  • Like
Reactions: lastb0isct and rtech
L

lastb0isct

New Member
Feb 12, 2018
24
2
3
38
MoMeanMugs said:
This is a vendor limitation imposed by SuperMicro. The crappy part is this was never listed on their product pages until well after the products were sold and in the wild. They've since updated their product pages to list the 16 drive limit. I've mentioned this in other threads.

OP, where are you located? I am willing to help, but I am about to leave town and won't be back for a week (need to travel the following week as well). You can send me a PM.
Click to expand...
Sent a conversation
 
You must log in or register to reply here.
Top Bottom