LSI 9206-16e (Dell TFJRW) not detecting drives or expanders

[Dave Corder]

Hi all,

I am upgrading from a whitebox build to a Dell R720 as the head unit for my FreeNAS server (well, TrueNAS now, but I'm behind on upgrades).

The short version of my issue is this: My Dell TFJRW card (low-profile PCIe card with 4x SFF-8644 ports, dual LSI SAS2308 controllers, flashed to IT mode 20.00.07.00) doesn't see any attached devices (not the expanders in my disk shelves, nor any disks).

Details:

I have four disk shelves based around SuperMicro and Norco chassis.
A SuperMicro 846 with the SAS2 backplane with built-in expander, connected to a SFF-8087 to SFF-8088 PCI bracket
Three other chassis use the Dell 6TDVN expander, connected to the drives
Most drives are SATA, but a handful are SAS
Chassis are attached with one SFF-8644 to SFF-8088 cable each (IBM/Amphenol cables, part number 2726182).
Disks and expanders are working fine - I hooked them back up to a 9201-16e card and everything showed up fine and the pools came back online in FreeNAS
I have the BIOS flashed to the R720's mini raid controller. Entering it during boot, I can see the mini controller and the 9201-16e, but the 9206-16e card doesn't show up there at all.
FreeNAS sees all the cards

Unfortunately, I was doing this upgrade swap yesterday while my wife and kids were out, and they needed their Plex up and running by the time they got back, so I didn't have much time to really troubleshoot the issue before just hooking everything up to the old 9201-16e card I was using in my old whitebox build.

Does anyone have any idea what the issue might be or suggestions on what to check the next time I have the time to troubleshoot? Could it be the cables (I assumed SFF-8644 to SFF-8088 was fairly generic, but maybe that's not the case).

Thanks!

 

[Dave Corder]

If anyone else stumbles upon this, it turned out to be due to an incomplete erase and a resulting mismatch in the NVDATA Version (Default) and NVDATA Version (Persistent). The clue was in this old Reddit thread that I stumbled upon while looking for something else.

Before:

➜  ~ sudo /tmp/sas2flash -list -c 2
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2308_2(D1)

    Controller Number              : 2
    Controller                     : SAS2308_2(D1)
    PCI Address                    : 00:45:00:00
    SAS Address                    : 5000d31-0-009e-4007
    NVDATA Version (Default)       : 14.01.00.06
    NVDATA Version (Persistent)    : 14.01.00.08
    Firmware Product ID            : 0x2214 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9206-16e
    BIOS Version                   : 07.39.02.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9206-16E
    Board Assembly                 : H3-25553-01A
    Board Tracer Number            : SV50501081

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.
➜  ~ sudo /tmp/sas2flash -list -c 3
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2308_2(D1)

    Controller Number              : 3
    Controller                     : SAS2308_2(D1)
    PCI Address                    : 00:47:00:00
    SAS Address                    : 5000d31-0-009e-400f
    NVDATA Version (Default)       : 14.01.00.06
    NVDATA Version (Persistent)    : 14.01.00.08
    Firmware Product ID            : 0x2214 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9206-16e
    BIOS Version                   : 07.39.02.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9206-16E
    Board Assembly                 : H3-25553-01A
    Board Tracer Number            : SV50501081

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

I booted @fohdeesha's PERC Mini crossflashing utils and did the following.

From FreeDOS:

sas2flsh -c 2 -o -e 6
sas2flsh -c 3 -o -e 6

Then I booted to the Linux LiveCD and ran this (taken from here):

addr=45:00.0
rmmod megaraid_sas mpt3sas mptctl mptbase
echo 16 > /proc/sys/vm/nr_hugepages
/root/lsirec/lsirec 0000:$addr unbind
/root/lsirec/lsirec 0000:$addr halt
/root/lsirec/lsirec 0000:$addr hostboot /root/H710/9207-8.bin
/root/lsirec/lsirec 0000:$addr rescan
sleep 30
# erase FW again now from lsiutil to be sure it's dell-free
/root/lsiutil/lsiutil -p3 -a 3,8, 33
# flash FW to onboard flash
/root/lsiutil/lsiutil -p3 -f /root/H710/9207-8.bin -y 2

addr=47:00.0
/root/lsirec/lsirec 0000:$addr unbind
/root/lsirec/lsirec 0000:$addr halt
/root/lsirec/lsirec 0000:$addr hostboot /root/H710/9207-8.bin
/root/lsirec/lsirec 0000:$addr rescan
sleep 30
# erase FW again now from lsiutil to be sure it's dell-free
/root/lsiutil/lsiutil -p4 -a 3,8, 33
# flash FW to onboard flash
/root/lsiutil/lsiutil -p4 -f /root/H710/9207-8.bin -y 2

I was lazy and just used the 9207-8i firmware instead of the "proper" 9206-16e firmware (which works, but my card shows up as a 9207-8i now in sas2flash). After setting the SAS addresses on both controllers, I was up and running and all is well.

LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2308_2(D1)

    Controller Number              : 3
    Controller                     : SAS2308_2(D1)
    PCI Address                    : 00:47:00:00
    SAS Address                    : 5000d31-0-009e-400f
    NVDATA Version (Default)       : 14.01.00.06
    NVDATA Version (Persistent)    : 14.01.00.06
    Firmware Product ID            : 0x2214 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9207-8i
    BIOS Version                   : N/A
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9207-8i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.