Netapp DS2246 & LSI 3008 IT Mode Issues in Windows 10 for Workstations

[thorondorwest]

I am getting a bit frustrated and figure perhaps I should engage the Hive Mind that is STH forums to see if anyone has an idea of how to fix this issue.

I have a Supermicro H11DSi-NT AMD EPYC based system with the integrated LSI 3008 SAS HBA connected to a NETAPP DS2246 disk shelf with 6 Seagate EXOS 2.4TB 10K RPM drives.

If I am using the IR firmware I am able to create RAID arrays in MegaRAID with no issues and they work just fine although I keep getting "imminent disk failure" messages that seem to randomly choose a slot to complain about, however the array itself functions just fine. Connecting the same drives directly to the SAS controller and I do not see this message so I am thinking it is an error related to the NAPP firmware and the 3008 having issues exchanging SMART data (kind of expected).

If, however, I cross-flash the chip to the IT mode firmware using either the Supermicro supplied version or the Broadcom 9300-8i the flash completes successfully, MegaRAID is able to see the disks and the enclosure, but the drives are not being passed through to Windows Disk Management/Storage Spaces.

If I connect the drives directly instead of via the shelf they appear in Windows Disk Management immediately even when the card is in IR mode. I do a lot of 4K video and need to use this shelf as an (expandable) media storage library for B roll footage, archived videos, etc. So I really don't want to use the built in RAID functionality because it limits me to flavors of RAID 0/1, which costs me way too much capacity. I am planning on using a storage spaces double parity pool as soon as my 7th drive arrives (one was defective and I am awaiting a replacement).

However before I can do that I need to figure out why the 3008 in IT mode is not passing the disks to windows if the shelf is hooked up but will pass them just fine if connected directly.

Any tips?

Thanks!

 

[gregsachs]

What happens if you set a drive to JBOD mode with IR firmware?
megacli -AdpSetProp EnableJBOD 1 -a0
Then need to explicitly set each drive to JBOD mode(will kill any raid using that drive). I have an intel MB with RMSCB8080 which is a 2208 based board that runs a mix of a Raid 1 volume for boot and JBOD devices used for storage spaces. That only worked when I went to P20 firmware, did not work with P19. I'm not sure what the equivalent is with the 3008 based boards. Note also that the 2246 is most likely a 6gb unit, (IOM6 or IOM12 installed???) and I wonder if could have a negotiation issue there; can you force the device(s) into 6gb mode?

 

[thorondorwest]

MegaCLI according to the readme file does not support any of the 3008 based/9300 series adapters. Good point I forgot to mention I have IOM6 installed and MegaRAID reports them successfully negotiating a 6Gbp/s link speed in the enclosure, but I cannot get them to show up in Disk Manager.

Do you think its worth trying MegaCLI even though it does not support SAS3 adapters?

What happens if you set a drive to JBOD mode with IR firmware?
megacli -AdpSetProp EnableJBOD 1 -a0
Then need to explicitly set each drive to JBOD mode(will kill any raid using that drive). I have an intel MB with RMSCB8080 which is a 2208 based board that runs a mix of a Raid 1 volume for boot and JBOD devices used for storage spaces. That only worked when I went to P20 firmware, did not work with P19. I'm not sure what the equivalent is with the 3008 based boards. Note also that the 2246 is most likely a 6gb unit, (IOM6 or IOM12 installed???) and I wonder if could have a negotiation issue there; can you force the device(s) into 6gb mode?

 

[thorondorwest]

What happens if you set a drive to JBOD mode with IR firmware?
megacli -AdpSetProp EnableJBOD 1 -a0
Then need to explicitly set each drive to JBOD mode(will kill any raid using that drive). I have an intel MB with RMSCB8080 which is a 2208 based board that runs a mix of a Raid 1 volume for boot and JBOD devices used for storage spaces. That only worked when I went to P20 firmware, did not work with P19. I'm not sure what the equivalent is with the 3008 based boards. Note also that the 2246 is most likely a 6gb unit, (IOM6 or IOM12 installed???) and I wonder if could have a negotiation issue there; can you force the device(s) into 6gb mode?

I tried MegaCLI and MegaCLI 64, the latest versions I can find are from 2014 and it fails to locate the adapter.

Is there an alternative for the 12Gb/s cards? I can't find anything on the Broadcom site.

 

[gregsachs]

Storcli looks like it
StorCLI - Thomas-Krenn-Wiki
Here is the manual
StorCLI Reference Manual (870 KB)
Commands are: (x=controller number, probably 0)
storcli /cx show jbod
storcli /cx set jbod=<on|off>
I think you want to do:
storcli64 show
to find the controller,
then
storcli /cx set JBOD=on
to enable on the controller, then
storcli64 /cx /dall show
to list disks,
then storcli /cx add jbod drives=eX:sX
using the list of disks...
StorCLI Reference Manual (870 KB)

 

[thorondorwest]

Running the following gives me:

storcli64 /c0 set JBOD=on

CLI Version = 007.0813.0000.0000 Dec 14, 2018
Operating system = Unknown
Controller = 0
Status = Failure
Description = Un-supported command

Same error when simply trying to show the JBODs. I'm wondering if I reflash again to IT mode and try this if it will work....

 

[thorondorwest]

Storcli looks like it
StorCLI - Thomas-Krenn-Wiki
Here is the manual
StorCLI Reference Manual (870 KB)
Commands are: (x=controller number, probably 0)
storcli /cx show jbod
storcli /cx set jbod=<on|off>
I think you want to do:
storcli64 show
to find the controller,
then
storcli /cx set JBOD=on
to enable on the controller, then
storcli64 /cx /dall show
to list disks,
then storcli /cx add jbod drives=eX:sX
using the list of disks...
StorCLI Reference Manual (870 KB)

CLI Version = 007.0813.0000.0000 Dec 14, 2018
Operating system = Unknown
Controller = 0
Status = Success
Description = None
Product Name = LSI3008-IR
Serial Number = 500304802484b500
SAS Address = 500304802484b500
PCI Address = 00:41:00:00
System Time = 02/26/2019 11:07:12
FW Package Build = 00.00.00.00
FW Version = 16.00.01.00
BIOS Version = 08.37.00.00_18.00.00.00
NVDATA Version = 14.01.49.39
Firmware Product ID = 0x2721
Driver Name = LSI_SAS3
Driver Version = 2.51.24.00
Bus Number = 65
Device Number = 0
Function Number = 0
Vendor Id = 0x1000
Device Id = 0x97
SubVendor Id = 0x15D9
SubDevice Id = 0x808
Board Name = LSI3008-IR
Board Assembly = N/A
Board Tracer Number = N/A
Physical Drives = 5
PD LIST :
=======
-----------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
-----------------------------------------------------------------------
5:0 0 UGood - 2.181 TB SAS HDD N N 512B ST2400MM0129 U
5:2 1 UGood - 2.181 TB SAS HDD N N 512B ST2400MM0129 U
5:3 2 UGood - 2.181 TB SAS HDD N N 512B ST2400MM0129 U
5:4 3 UGood - 2.181 TB SAS HDD N N 512B ST2400MM0129 U
5:5 4 UGood - 2.181 TB SAS HDD N N 512B ST2400MM0129 U
-----------------------------------------------------------------------
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
Requested Boot Drive = Not Set

 

[gregsachs]

I got nothing. Got a different HBA to try with? The 12gb/s is no benefit with a 2246.

 

[thorondorwest]

I appreciate the help. This is going to be a project I think. Might have to track down some Xyratec controllers to toss in the shelf to get away from the Netapp strangeness. I am using my onboard SAS controller (on the Mobo) via a 8087/8088 pass through to a QSFP+ cable. I don't have any spare PCI-E slots on this board so I am likely going to have to get creative. Starting with cross referencing compatible controllers which I think also may include Dell Compellant.

 

[gregsachs]

Couple other ideas; 1 would be to boot ubuntu or similar and see if the drives show there; I think that controller may only be supported under server, not win 10.
In theory the controllers have all the smarts and the expander, so the enclosure is just a dumb backplane(reference, Storage bridge bay spec); and I have heard of people using a xyratex/compellant/etc unit in place of netapp. Can get a dell unit for $60,
Pure Storage / Dell HB-SBB2-E601 / 0956788-03 6Gbps SAS EBOD Controller 658759139313 | eBay
2: Are you using both IOM or just one? Trying to multipath? For SATA drives, I have seen them only show up on one IOM. If you have sata drives that only show on one IOM, and are trying to m-p, I'd think wierdness might happen.

 

[thorondorwest]

Couple other ideas; 1 would be to boot ubuntu or similar and see if the drives show there; I think that controller may only be supported under server, not win 10.
In theory the controllers have all the smarts and the expander, so the enclosure is just a dumb backplane(reference, Storage bridge bay spec); and I have heard of people using a xyratex/compellant/etc unit in place of netapp. Can get a dell unit for $60,
Pure Storage / Dell HB-SBB2-E601 / 0956788-03 6Gbps SAS EBOD Controller 658759139313 | eBay
2: Are you using both IOM or just one? Trying to multipath? For SATA drives, I have seen them only show up on one IOM. If you have sata drives that only show on one IOM, and are trying to m-p, I'd think wierdness might happen.

I have two cables and two IOMs, I will try different cabling scenarios and see if that makes a difference. I have already tried using only a single cable but I think I didn't try switching that single cable to the other controller. When I used two cables into a single IOM6 it sees it as a single controller. If I plug a cable into each IOM it sees it as two enclosures inside of MEGARAID.

The DS2246 only supports SAS drives, I am using 6 Seagate Exos 2.4TB 12Gb/s SAS drives currently. This weekend I will give Linux a try.

 

[gregsachs]

I have two cables and two IOMs, I will try different cabling scenarios and see if that makes a difference. I have already tried using only a single cable but I think I didn't try switching that single cable to the other controller. When I used two cables into a single IOM6 it sees it as a single controller. If I plug a cable into each IOM it sees it as two enclosures inside of MEGARAID.

The DS2246 only supports SAS drives, I am using 6 Seagate Exos 2.4TB 12Gb/s SAS drives currently. This weekend I will give Linux a try.

I've put sata drives in mine; but they only show on one IOM. I suspect I would need different tray/interposers to make them show on both.
I couldn't find any reference to the 3008 IT or IR supporting multipath. There is a google reference that I found with a broken link that suggests IT and IR do not support multipath
I'd start by simplifying as much as possible, try it mode with a single cable to single IOM. Start with the left hand IOM as you face the back, that is A, and connect to the square port.
See:
https://library.netapp.com/ecm/ecm_get_file/ecmm1280301
(you may already know that, apologies)

 

[thorondorwest]

Here is a dump from StorCLI. Notice the "support JBOD=No" under the controller properties? Digging around on the Avago website this appears to be a bug that was fixed some time ago in the OEM firmware. I am wondering if SuperMicro is crippling the chip to sell more backplanes? I will give Linux a shot this weekend. I have it cabled to the square port on the left side facing the back and am getting the same behavior.

Is yours working under Windows Server? If so what version of the OS are you running? I am using Windows 10 for Workstations which may have had MS cripple the ability access a SAS expansion shelf because it also supports SMB Direct, which would make it an awfully cheap file server if setup with a JBOD. I have also tried the LSI Storage Authority and somewhat similar to StorCLI I am only able to access a very small portion of the advertised functionality. LSA even reports it as an SES expansion shelf, sees everything, etc.

Do you see anything strange in the dump below?

V/r,

Tom


storcli /c0 show all
CLI Version = 007.0912.0000.0000 Dec 27, 2018
Operating system = Unknown
Controller = 0
Status = Success
Description = None

Basics :
======
Controller = 0
Adapter Type = SAS3008(C0)
Model = LSI3008-IT
Serial Number = 500304802484b500
Current System Date/time = 02/27/2019 15:21:31
Concurrent commands supported = 3072
SAS Address = 500304802484b500
PCI Address = 00:41:00:00
Mfg Date = 00/00/00
Rework Date = 00/00/00
Revision No = N/A

Version :
=======
Firmware Package Build = 00.00.00.00
Firmware Version = 16.00.01.00
Bios Version = 08.37.00.00_18.00.00.00
NVDATA Version = 14.01.48.40
Driver Name = LSI_SAS3i
Driver Version = 2.51.24.80

PCI Version :
===========
Vendor Id = 0x1000
Device Id = 0x97
SubVendor Id = 0x15D9
SubDevice Id = 0x808
Host Interface = PCIE
Device Interface = SAS-12G
Bus Number = 65
Device Number = 0
Function Number = 0

Pending Images in Flash :
=======================
Image name = No pending images

Status :
======
Controller Status = OK
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
Bios was not detected during boot = No
Controller has booted into safe mode = No

Supported Adapter Operations :
============================
Alarm Control = No
Cluster Support = No
Self Diagnostic = No
Deny SCSI Passthrough = No
Deny SMP Passthrough = No
Deny STP Passthrough = No
Support more than 8 Phys = No
FW and Event Time in GMT = No
Support Enclosure Enumeration = Yes
Support Allowed Operations = Yes
Support Multipath = Yes
Support Security = No
Support Config Page Model = No
Support the OCE without adding drives = No
support EKM = No
Snapshot Enabled = No
Support PFK = No
Support PI = No
Support Shield State = No
Support Set Link Speed = No
Support JBOD = No
Disable Online PFK Change = No
Real Time Scheduler = No
Support Reset Now = No
Support Emulated Drives = No

HwCfg :
=====
ChipRevision = C0
BatteryFRU = N/A
Front End Port Count = 1
Backend Port Count = 8
Serial Debugger = Absent
NVRAM Size = 0KB
Flash Size = 32MB
On Board Memory Size = 0MB
On Board Expander = Absent
Temperature Sensor for ROC = Present
Temperature Sensor for Controller = Absent
Current Size of CacheCade (GB) = 0
Current Size of FW Cache (MB) = 0
ROC temperature(Degree Celcius) = 51

Policies :
========
Policies Table :
==============
------------------------------------------------
Policy Current Default
------------------------------------------------
Predictive Fail Poll Interval 0 sec
Interrupt Throttle Active Count 0
Interrupt Throttle Completion 0 us
Rebuild Rate 0 % 30%
PR Rate 0 % 30%
BGI Rate 0 % 30%
Check Consistency Rate 0 % 30%
Reconstruction Rate 0 % 30%
Cache Flush Interval 0s
------------------------------------------------
Flush Time(Default) = 4s
Drive Coercion Mode = none
Auto Rebuild = Off
Battery Warning = Off
ECC Bucket Size = 0
ECC Bucket Leak Rate (hrs) = 0
Restore HotSpare on Insertion = Off
Expose Enclosure Devices = Off
Maintain PD Fail History = Off
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = None
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off

Boot :
====
Max Drives to Spinup at One Time = 0
Maximum number of direct attached drives to spin up in 1 min = 0
Delay Among Spinup Groups (sec) = 0
Allow Boot with Preserved Cache = On

Defaults :
========
Phy Polarity = 0
Phy PolaritySplit = 0
Cached IO = Off
Default spin down time (mins) = 0
Coercion Mode = None
ZCR Config = Unknown
Max Chained Enclosures = 0
Direct PD Mapping = No
Restore Hot Spare on Insertion = No
Expose Enclosure Devices = No
Maintain PD Fail History = No
Zero Based Enclosure Enumeration = No
Disable Puncturing = No
Un-Certified Hard Disk Drives = Block
SMART Mode = Mode 6
Enable LED Header = No
LED Show Drive Activity = No
Dirty LED Shows Drive Activity = No
EnableCrashDump = No
Disable Online Controller Reset = No
Treat Single span R1E as R10 = No
Power Saving option = Enable
TTY Log In Flash = No
Auto Enhanced Import = No
Enable Shield State = No
Time taken to detect CME = 60 sec

Capabilities :
============
Supported Drives = SAS, SATA
Boot Volume Supported = NO
Enable JBOD = Not Allowed
Max Parallel Commands = 3072
Max SGE Count = 128
Max Data Transfer Size = 32 sectors
Max Strips PerIO = 0
Max Configurable CacheCade Size = 0
Min Strip Size = 512Bytes
Max Strip Size = 512Bytes
Scheduled Tasks = NA
Enclosure Information :
=====================
-----------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM ProdID VendorSpecific
-----------------------------------------------------------------
6 OK 24 6 2 4 8 0 2 DS224IOM6 PMCSIERA
-----------------------------------------------------------------

 

[gregsachs]

remember I have 2208 based cards, not 3008. My ds2246 is currently shutdown-i was using before I had a server with 2.5" bays, but I can easily power it back up. I am running H-V 2016, using storage spaces.

C0 is mezzanine card, has JBOD and Raid config.
C1 is pure JBOD.
The JBOD box is a xyratex HB-1235 with 6gb ioms.

 

[thorondorwest]

Supermicro has terrible customer service the tech I have been working with is grossly incompetent, combative, and has some issues with understanding English. I think I have narrowed the issue down to this:

Expose Enclosure Devices = Yes

Shows on your C1 card, while it is listed as = No on my card and your C0 config. The SAS3IRCU utility has no ability to change this setting and both MegaCLI and StorCLI spit back an "unsupported command" when I try to modify it. It appears to be a default setting in the firmware provided by SM that prevents enumeration of drives in an expansion shelf which is why it will not pass them off to Windows. I will need to get SM to modify their firmware in order for this to work. This explains why I can create a SW RAID in IR mode and everything works as expected but the disks will not show up in IT mode when behind an expansion shelf.

remember I have 2208 based cards, not 3008. My ds2246 is currently shutdown-i was using before I had a server with 2.5" bays, but I can easily power it back up. I am running H-V 2016, using storage spaces.

C0 is mezzanine card, has JBOD and Raid config.
C1 is pure JBOD.
The JBOD box is a xyratex HB-1235 with 6gb ioms.

 

[thorondorwest]

From your C1 config:

Flush Time(Default) = 4s
Drive Coercion Mode = none
Auto Rebuild = On
Battery Warning = On
ECC Bucket Size = 15
ECC Bucket Leak Rate (hrs) = 24
Restore HotSpare on Insertion = Off
Expose Enclosure Devices = On
Maintain PD Fail History = On
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = Auto
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off

From my config and also your C0 config:

Flush Time(Default) = 4s
Drive Coercion Mode = none
Auto Rebuild = Off
Battery Warning = Off
ECC Bucket Size = 0
ECC Bucket Leak Rate (hrs) = 0
Restore HotSpare on Insertion = Off
Expose Enclosure Devices = Off
Maintain PD Fail History = Off
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = None
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off

 

[gregsachs]

Glad I was able to help you track down the issue; good luck getting it worked out. Is there generic firmware that can be flashed?

 

[thorondorwest]

Unfortunately not for the embedded 3008 based controllers - I have to have SuperMicro provide me with a custom firmware.

 

[thorondorwest]

Supermicro has refused to assist in troubleshooting. This is the last Supermicro product I purchase as their support is absolutely horrendous. They ran me around in circles for months wasting my time and then suggested that I contact sales to purchase one of their "certified compatible" server enclosures and closed my case.

Glad I was able to help you track down the issue; good luck getting it worked out. Is there generic firmware that can be flashed?

There has to be a way to modify these settings.

 

[wheresmymedia]

I know that this is an old thread, but I am having the exact same issue with a lenovo 2225 (LIS 3008e) flashed to P16 IT Mode with a DS4246. I haven't tried IR mode as the card is a pure HBA stock. I think they you came to the wrong conclusion about " Expose Enclosure Devices = Off" as we are talking about IT Mode. Most of these options normally on IR flashed cards won't show up in IT mode and as such the capability would show up as no, because there isn't a purpose to the setting on an HBA. Anyway, I'm still banging my head against the wall trying to get this to work with windows. The controller definitely sees everything with no issue. Did you ever make any progress?

 

[wheresmymedia]

Well, I tried installing TrueNas 12 (FreeNas) on a Hyper-V VM while passing through the lenovo 2225 (LIS 3008e) to the VM and it works in TrueNs. So it is definitively an OS or Windows Driver related issue. So I guess this configuration in Server 2016 is out! Curious if it works in Server 2019.