Dell R720xd cannot see disks in NetApp DS4246

[MuscleLazy]

I have a Dell R720xd with a PERC H710 Mini flashed to IT mode and TrueNAS Scale 22.02.3 running on it. I purchased a NetApp DS4246 with dual IOM6's and dual power supplies. I had a PERC H810 flashed to IT mode and used a SFF-8436 to SFF-8088 cable, but the SATA disks are not seen in TrueNAS Scale.

So far I tried 2 cables and 3 different cards, with no success:

• Dell PERC H810 flashed to IT mode with @fohdeesha's instructions
• Dell PERC H200E (LSI 9200-8E)
• HP H221 (LSI SAS9207-8e)

Right now I have installed the two flashed cards, PERC H710 mini (internal) and PERC H810 (external). The cards are detected properly by Scale (ADDR0 is the internal PERC H710, with all internal disks in use by a pool):

# lspci | grep LSI
02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
44:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2308_2(D1)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2308_2(D1)   20.00.07.00    14.01.00.06    07.39.02.00     00:02:00:00
1  SAS2308_2(D1)   20.00.07.00    14.01.00.06    07.39.02.00     00:44:00:00

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

I use the same disk model, for both Dell R720xd and NetApp DS4246: HGST Ultrastar He8 Helium (HUH728080ALE601).

The NetApp ID is set to 00 and cable is connected into square port as per this diagram. BTW, disks are installed correctly, flush with the bottom of each caddie. I also installed one interposer for one disk, no dice. When I tested the connectivity, I had both devices shutdown, power first DS4246, wait until fans slow down then power R720xd.

Can anyone using these appliances post some guidance what to troubleshoot next? I'm running out of ideas, thank you for your help.

Update: The issue is resolved. Out of the three cards, only PERC H810 was functional, the other two cards were sold defective or badly flashed by eBay seller. Also, it turns out Debian cares about the firmware present into PERC H810 controller, which had 9207-8i firmware flashed into. As soon as I flashed the 9207-8e firmware, everything was functional.



There is something interesting related how the enclosure disks are seen by Debian. If I connect the cable into top “square” IOM6 port, all disks are seen, no matter where they are installed, with or without interposers installed. If I connect the cable into bottom “square” IOM6 port, only the disks with interposer installed are seen by Debian.

Each of the two IOM6 controllers have one SAS path wired to them. Without an interposer, the SATA devices have no second path. Adding the interposer results in both paths being present.

 

[mobilenvidia]

Disks are probably formatted as 520byte sectors vs 512byte which is what us mortals and SAS controllers work with
You'll need to reformat each driver to 512byte sector size if that is possible, there are posts in HDD re this very thing

 

[MuscleLazy]

The HGST disks were purchased from same vendor on Amazon, 12 were used into R720xd and 12 into DS4246. The disks were instantly detected in R720xd. I will still check the formatting but I'm pretty sure that's not the case. @mobilenvidia, if I take a working disk offline from a pool VDEV and replace it with one of the disks installed into DS4246, should be considered a good test that all disks are formatted correctly, if disk shows as available in R720xd? Link to 512b formatting guide.

 

[i386]

dual IOM6'

is the hba connected to the correct iom? with sata (and without interposers) only one iom could "see" the disks

 

[Stephan]

Can you see the enclosure at all? "lsscsi -g". Then try "sg_ses /dev/sgXX" to see if you can talk to it. Also "sg_ses -R --page=es /dev/sgXX" or "sg_ses -R --page=aes /dev/sgXX". ES is enclosure status. Worry about disks later. Swap controllers from bottom to top for good measure. Also swap power supplies in their places.

 

[MuscleLazy]

only one iom could "see" the disks

Can you please give more details? I tried every possible combination, right now I have 4 disks at the top shelf and 8 disks at the bottom:

Disk with interposer	Disk	Disk	Disk
x	x	x	x
x	x	x	x
x	x	x	x
Disk	Disk	Disk	Disk
Disk	Disk	Disk	Disk

I tried connecting on each of both "square" marked ports, using the proper boot sequence (all hardware powered off, power DS4246 and wait until fans slow down, power R720xd).

Can you see the enclosure at all? "lsscsi -g"

I'm running TrueNAS Scale, which is Debian bullseye 11. lsscsi is unfortunately not available, sg_ses is available. Could the enclosure be seen in lspci? If yes, can you please point what should I grep for, or maybe an alternate Debian command I can use?

 

[mobilenvidia]

Boot to Linux on a USB and diagnose from there

 

[MuscleLazy]

Boot to Linux on a USB and diagnose from there

Will do right away and report back, did not think about it.

Edit: @Stephan lsscsi -g shows only the R720xd disks, how would the enclosure show into list?

root@debian:~# lsscsi -g
[0:0:0:0]    disk    ATA      HUH728080ALE601  0003  /dev/sda   /dev/sg0
[0:0:1:0]    disk    ATA      HUH728080ALE601  0001  /dev/sdb   /dev/sg1
[0:0:2:0]    disk    ATA      HUH728080ALE601  0003  /dev/sdc   /dev/sg2
[0:0:3:0]    disk    ATA      HUH728080ALE601  0001  /dev/sdd   /dev/sg3
[0:0:4:0]    disk    ATA      HUH728080ALE601  0003  /dev/sde   /dev/sg4
[0:0:5:0]    disk    ATA      HUH728080ALE601  0001  /dev/sdf   /dev/sg5
[0:0:6:0]    disk    ATA      HUH728080ALE601  0001  /dev/sdg   /dev/sg6
[0:0:7:0]    disk    ATA      HUH728080ALE601  0001  /dev/sdh   /dev/sg7
[0:0:8:0]    disk    ATA      HUH728080ALE601  0001  /dev/sdi   /dev/sg8
[0:0:9:0]    disk    ATA      Samsung SSD 870  1B6Q  /dev/sdj   /dev/sg9
[0:0:10:0]   disk    ATA      Samsung SSD 870  1B6Q  /dev/sdk   /dev/sg10
[0:0:11:0]   disk    ATA      HUH728080ALE601  0001  /dev/sdl   /dev/sg11
[0:0:12:0]   disk    ATA      HUH728080ALE601  0001  /dev/sdm   /dev/sg12
[0:0:13:0]   disk    ATA      HUH728080ALE601  0003  /dev/sdn   /dev/sg13
[2:0:0:0]    disk    SanDisk  Cruzer Glide     1.00  /dev/sdp   /dev/sg15
[3:0:0:0]    disk    Lexar 128GB SSD           0     /dev/sdo   /dev/sg14

By default I'm booting from UEFI, switching temporarily to BIOS boot does not do anything.

# sg_scan -i
/dev/sg0: scsi0 channel=0 id=0 lun=0
    ATA       HUH728080ALE601   0003 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg1: scsi0 channel=0 id=1 lun=0
    ATA       HUH728080ALE601   0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg2: scsi0 channel=0 id=2 lun=0
    ATA       HUH728080ALE601   0003 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg3: scsi0 channel=0 id=3 lun=0
    ATA       HUH728080ALE601   0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg4: scsi0 channel=0 id=4 lun=0
    ATA       HUH728080ALE601   0003 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg5: scsi0 channel=0 id=5 lun=0
    ATA       HUH728080ALE601   0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg6: scsi0 channel=0 id=6 lun=0
    ATA       HUH728080ALE601   0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg7: scsi0 channel=0 id=7 lun=0
    ATA       HUH728080ALE601   0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg8: scsi0 channel=0 id=8 lun=0
    ATA       HUH728080ALE601   0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg9: scsi0 channel=0 id=9 lun=0
    ATA       Samsung SSD 870   1B6Q [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg10: scsi0 channel=0 id=10 lun=0
    ATA       Samsung SSD 870   1B6Q [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg11: scsi0 channel=0 id=11 lun=0
    ATA       HUH728080ALE601   0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg12: scsi0 channel=0 id=12 lun=0
    ATA       HUH728080ALE601   0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg13: scsi0 channel=0 id=13 lun=0
    ATA       HUH728080ALE601   0003 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg14: scsi2 channel=0 id=0 lun=0
    Lexar 128GB SSD.            0    [rmb=0 cmdq=1 pqual=0 pdev=0x0]

I don't know what that ID16 disk is, it should not exist.

 

[gregsachs]

With SATA disks and no interposer, only one IOM is actually connected to the disks.
Sg_Scan --scsi --scsi should show a line like this:
SCSI1:0,67,0 claimed=0 pdt=dh NETAPP DS224IOM6 0173
for the enclosure
Get that to show up first.
SBB "A" is the top slot, I would expect that one to connect to SATA disks.

https://library.netapp.com/ecm/ecm_get_file/ECMP1115547

Also, super crazy idea, but the netapp/xyratex caddies have two mounting positions-one is biased to the front, for use with interposers. If you mount the disks in that set of holes, they don't connect to the shelf. I know someone had that issue recently.
Edit: ^^^I see you covered that idea
(and if you are using a DS2246, like I have, the left hand IOM is the one that can see SATA disks)

 

[Stephan]

lsscsi will report "enclosu" instead of "disk" in 2nd column. Maybe wrecked NetApp controllers since you already tested different SAS cards and two cables. Can you photograph year on PCB of mid-plane? Maybe they sold you a frankenstein DS4243 which does not like IOM6 controllers. Lightning damage, many ways to do damage. An alternative 6 GBps controller would be the HB-SBB2-E601-COMP. Do drive lights come on after power-on? If not, would smell like controller trouble.

 

[MuscleLazy]

Can you photograph year on PCB of mid-plane?

I presume the PCB year cannot be easily exposed, is it visible if I remove the caddies? I apologize, I'm not familiar. My enclosure is mounted against the wall. Disk lights are fine, both power supply lights are green.

 

[mobilenvidia]

Is 'slot 2' the Mini H710 HBA with a bootable drive set ?, meaning there is a BIOS on the HBA ?

Got a SAS drive you can try ?

I'm leaning to towards Stephan smelling a rat with the DS4246 controller

 

[Stephan]

Power down, pull every controller out and also all shrouds (PSU/controllers) and PSUs. Careful, only grip onto metal once a piece is unlocked otherwise you will break off plastic latches. Shine lamp into inside, there should be some year code visible, and other numbers. Mid-plane should be 2009 or newer. Or I forget and this was visible from front with HDDs removed. Can't shutdown my own right now to check. If you can snap some high res pictures I can take a look. Are you in USA? If yes you are lucky, you can buy one of the Compellent controllers cheap, to check if they sold you a dud. ebay 20 bucks. Would also rid you of QSFP cable, those have SFF-8088 so normal rectangled SAS.

 

[MuscleLazy]

Is 'slot 2' the Mini H710 HBA with a bootable drive set?

The internal USB is connected to a 128GB Lexar SSD which hosts the TrueNAS Scale OS:

/dev/sg14: scsi2 channel=0 id=0 lun=0
    Lexar 128GB SSD.            0    [rmb=0 cmdq=1 pqual=0 pdev=0x0]

I don't have a SAS drive to test, unfortunately. I have no idea what that Slot 2 disk is, it shows every time I do a BIOS boot, with or without DS4246 enclosure powered. All R720xd disks showed correctly when I first installed Scale with DS4246 disconnected. After Scale install, I installed the flashed PERC H810 and connected the DS4246.

When I boot, I see an Avago section:

Power down, pull every controller out and also all shrouds (PSU/controllers) and PSUs.

Yes I can purchase items in US. I removed all components behind and looked at the board, there is no year anywhere, except 2 small while labels with serial numbers on it.



However, year 2011 is printed on the unit side. PSUs are also year 2011, IOM6s are year 2016.

 

[mobilenvidia]

To speed up boot = removing the MPT BIOS, no need for BIOS as you are booting to USB

Clear the H710 Mini firmware completely, will also remove both BIOS's
Flash the IT Firmware only
Thats it reboot
This has nothing to do with DS4246 issue you are having but will make life quicker with no doubt endless reboots and waiting for MPT BIOS to load and find drives

If anything maybe flash the MPT BIOS to the H810 in IT mode and see what is says when machine boots it should do the same when it detects the drives on DS4246, would be more usefull in diagnosing
When H810 doesn't see drives at boot then OS won't either
Then do the same as above wipe FW then flash IT FW only once issue resolved

 

[MuscleLazy]

To speed up boot = removing the MPT BIOS, no need for BIOS as you are booting to USB

Yes, I'm familiar with it, @fohdeesha explained into his guide. Originally I did not have the x86-BIOS installed, but I prefer to have it in both cards.

If anything maybe flash the MPT BIOS to the H810 in IT mode

I have x86-BIOS installed in both PERCs. This is the server BIOS output, both cards are seen, but only H710 sees the disks:

 

[mobilenvidia]

Take the H710 out of machine for diagnosing real issue (or remove its BIOS as it cluttering up boot)
The H810 BIOS should now show at boot with its BIOS and nothing else to confuse matters

 

[MuscleLazy]

Take the H710 out of machine for diagnosing real issue

Will do and report back in an hour. Thank you all for the troubleshooting support, is much appreciated.

Edit: No disks showing, only PERC H810:



Disks without and with interposer installed:

 

[mobilenvidia]

Go into the R810 BIOS and have a poke about
What can you see in here ?

Can it at least see the DS4246 ? or an enclosure ?

 

[MuscleLazy]

Go into the R810 BIOS and have a poke about

I cannot go into Avago configuration (CTRL + C) with flashed H810, no matter what I do. I hope that is what you mean by R810 BIOS, I'm sorry I'm really not good at it. I get the message that the configuration will load after initialization, the initializing firmware message shows, then it proceeds to the TrueNAS Scale loading. Maybe @fohdeesha can bring some light into this? I changed the boot to BIOS from UEFI, same result.



Edit: In order to get into Avago configuration, both cards need to be installed, having only the H810 PCIe card will not load the utility. With both cards installed, I get this menu:



I changed the display mode to:



After reboot I see: