2x LSI 9211-8i IT no HDD

[LunchB0xK1ller]

I am very new to this forum but I need some guidance here if anyone could help. A few months ago, I purchased 2 LSI 9211-8I controllers that were pre-flashed to IT mode with P19 and P20 firmware on them. I have been using one of the controllers on and off for the last few months and the second one sat in the drawer until my new chassis was ready. This is where it begins.


I received my enclosure and all the parts needed to assemble it finally and I tested all components before assembly, everything worked. I installed the components into the chassis, I removed the 9211 card that I have been using from the old system and installed it, put the chassis in my rack and installed all of the drives into their sleds to go in the hot swap bays. On first powerup I was successfully able to boot into the OS, which is UnRaid, but none of the drives showed. I rebooted the system and could see the activity lights on the backplane are working but when I hit the OS no HDDs again.


I immediately thought maybe I didn’t plug something in so I removed the system from the rack and sat it on my test bench desk and opened the top to look, everything was plugged in. I went ahead and double checked everything anyway and once I was done I gave it power and internet and booted. Once booted into the OS I showed all of the drives and it was working great. I powered it down and installed in back into the rack and booted it up to find no drives again.


I did this a few times and when I took it out it worked but installed didn’t work and now it just doesn’t show drives at all. I installed the second card I had into a different system to first test the card that card worked fine until the first reboot, now no drives show. I put both card into different systems and they both show power through the LEDs on the cards but no HDDs show and no activity lights.

I have tried changing the cables to known good cables, new cables and the cables that came with the cards. I have tried known good breakout cables as well and no change. I even tried then in FreeNas, Proxmox, ESXI and CentOS.


I have tested these cards in the following systems,

Supermicro 825TQ with a Supermicro X9SRL-F ATX Server Motherboard and a E5-2665 with 16GB of ECC ram

Supermicro 836TQ with a Intel S5500BC motherboard dual x5675 CPUs 32GB ECC ram

Intel 2U chassis with a cp2600 board (not sure exact number on it) dual E5-2640s 64GB ECC ram

Norco JBOD chassis (no backplane, SATA breakout cables) Intel S2600CP2J Motherboard Dual E5-2670 SR0KX 64GB ECC ram


Sorry for the long post. Any ideas would be appreciated and welcomed. I am not sure where to go from here aside from reflashing the cards to see if that could help.

 

[BLinux]

i think it would be helpful for others to help you if you provide a little bit more low level information. so far, you've stated that the cards are either in working state or not showing HDDs. there's a lot of ground in between. for example:

0) does the heartbeat LED on the card blink when you power on the system (this confirms firmware is running)
1) if they have a BIOS ROM on them, do you see the card's ROM loading during POST?
2) if yes to #1, if you get into the BIOS of the card, do you see your HDDs?
3) once an OS is loaded, does your OS detect a device in the PCI-E slot the card is installed in?
4) does your OS load the driver for the HBA? if so, are there any error messages while loading the driver?
5) if the driver loads okay, does the driver see the storage devices? (even if it does, it doesn't mean your HDDs will show up. for example, if the sector size is unsupported, the driver will detect the storage devices but not present them)

 

[LunchB0xK1ller]

i think it would be helpful for others to help you if you provide a little bit more low level information. so far, you've stated that the cards are either in working state or not showing HDDs. there's a lot of ground in between. for example:

0) does the heartbeat LED on the card blink when you power on the system (this confirms firmware is running)
1) if they have a BIOS ROM on them, do you see the card's ROM loading during POST?
2) if yes to #1, if you get into the BIOS of the card, do you see your HDDs?
3) once an OS is loaded, does your OS detect a device in the PCI-E slot the card is installed in?
4) does your OS load the driver for the HBA? if so, are there any error messages while loading the driver?
5) if the driver loads okay, does the driver see the storage devices? (even if it does, it doesn't mean your HDDs will show up. for example, if the sector size is unsupported, the driver will detect the storage devices but not present them)

0) yes the heartbeat led is on and green
1) neither have a BIOS ROM on them
2) N/A
3) In BIOS I don't see the card but in the OS it shows a PCIE device
4) Well it was working before in the OS when it was on the test bench and I have no errors in my log in UnRaid
5) The drives are all filled with data from my storage server prior. With UnRaid in the past when I change hardware I just transferred the drives over and the USB boot disk and it picks up like nothing happened. This worked when it was on the bench and I initially installed the card first there and ran it.

 

[BLinux]

Ok. so we can skip 0-2. Let's examine 3-5 a little closer:

3) you say it shows in the OS. I don't know unraid internals that well, but I understand it is Linux based right? so, if you can login on a shell with root privileges, can you run something like:

# lspci | grep -i sas
07:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

Above is just example on my workstation with a LSI 3008 HBA; yours should show SAS2008. If so, can you take the PCI number (in my example 07:00.0) and show more detailed output like:

# lspci -s 07:00.0 -vvv

Of course, change "07:00.0" to what is on your system for the SAS2008 HBA. Can you post that output?

4) the SAS2008 in Linux, should be using the mpt2sas driver. So, after booting up, can you show the output to:

# dmesg |grep mpt2sas

5) are your HDDs SATA or SAS? if they are SATA, in Linux, it should show up under ata driver, so show the output of something like this:

# dmesg | grep ata

Also, can you show the output of:

# lsscsi

 

[LunchB0xK1ller]

Ok. so we can skip 0-2. Let's examine 3-5 a little closer:

3) you say it shows in the OS. I don't know unraid internals that well, but I understand it is Linux based right? so, if you can login on a shell with root privileges, can you run something like:

# lspci | grep -i sas
07:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

Above is just example on my workstation with a LSI 3008 HBA; yours should show SAS2008. If so, can you take the PCI number (in my example 07:00.0) and show more detailed output like:

# lspci -s 07:00.0 -vvv

Of course, change "07:00.0" to what is on your system for the SAS2008 HBA. Can you post that output?

4) the SAS2008 in Linux, should be using the mpt2sas driver. So, after booting up, can you show the output to:

# dmesg |grep mpt2sas

5) are your HDDs SATA or SAS? if they are SATA, in Linux, it should show up under ata driver, so show the output of something like this:

# dmesg | grep ata

Also, can you show the output of:

# lsscsi

As soon as I get home tonight I will get the above and get it posted for you. In advance I do appreciate everything.

 

[LunchB0xK1ller]

Ok. so we can skip 0-2. Let's examine 3-5 a little closer:

3) you say it shows in the OS. I don't know unraid internals that well, but I understand it is Linux based right? so, if you can login on a shell with root privileges, can you run something like:

# lspci | grep -i sas
07:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

Above is just example on my workstation with a LSI 3008 HBA; yours should show SAS2008. If so, can you take the PCI number (in my example 07:00.0) and show more detailed output like:

# lspci -s 07:00.0 -vvv

Of course, change "07:00.0" to what is on your system for the SAS2008 HBA. Can you post that output?

4) the SAS2008 in Linux, should be using the mpt2sas driver. So, after booting up, can you show the output to:

# dmesg |grep mpt2sas

5) are your HDDs SATA or SAS? if they are SATA, in Linux, it should show up under ata driver, so show the output of something like this:

# dmesg | grep ata

Also, can you show the output of:

# lsscsi

3-5) so now it doesn't show in the OS when I booted in and typed the command it did nothing at all there was no output.

only thing that shows anything is
# lspci | grep SATA
and it only shows the on board SATA controller.

I tried this through putty into my freenas install

 

[BLinux]

@LunchB0xK1ller can we just stick to Linux for the troubleshooting? not sure why you're now running freenas... the driver in freenas is different than the linux one, i think it is called 'mps' or something like that instead of 'mpt2sas'. i don't dig under the hood on freebsd/freenas often enough to remember exactly. the instructions above are meant for Linux; i figured you can ssh into your unraid, no?

 

[chinesestunna]

To possibly eliminate hardware faults have you tried to restore to original config with your old server and do any drives pick up? Sounds like your unRAID is installed on an USB drive and data drives are attached to the SAS controller? Please give us more details of the original build and what you are migrating to for a clearly picture and help eliminate problem points

 

[LunchB0xK1ller]

@LunchB0xK1ller can we just stick to Linux for the troubleshooting? not sure why you're now running freenas... the driver in freenas is different than the linux one, i think it is called 'mps' or something like that instead of 'mpt2sas'. i don't dig under the hood on freebsd/freenas often enough to remember exactly. the instructions above are meant for Linux; i figured you can ssh into your unraid, no?

I did ssh into my UnRaid install first and the outcome was the same. I figured I would try in freenas to see if there is a different outcome and there wasn't. I will do this again in UnRaid tonight through a ssh terminal and post the outcome here that way I can follow your guidance directly.

To possibly eliminate hardware faults have you tried to restore to original config with your old server and do any drives pick up? Sounds like your unRAID is installed on an USB drive and data drives are attached to the SAS controller? Please give us more details of the original build and what you are migrating to for a clearly picture and help eliminate problem points

The original build included all of the same components running on a open test bench. It was the Supermicro X9SRL-F ATX Server Motherboard and a E5-2665 with 16GB of ECC ram on a bench with a sas to sata breakout cable for the inital setup of the drives. I then moved the components into the enclosure, a sc825TQ, changing nothing along the way. I even left the USB with the OS in the board as it had a type a internal header. I plugged the sas to sata cable into the back plane of the case. That when the drives didn't show. I removed the system from my rack and verified all connections and the drives then showed, but only until I restarted it or it sat idle for more than a few moments.

The only thing that differed from the bench setup to the final setup is the addition of the back plane. I did try taking that out of the equation and using the sas to sata cables directly to the drives again and the same thing happened. they showed but then went away.

 

[kapone]

Have you tried different slots in the motherboard?

 

[LunchB0xK1ller]

Have you tried different slots in the motherboard?

Yes, I have tried each slot on 4 different boards so far.

 

[kapone]

Yes, I have tried each slot on 4 different boards so far.

Any chance there was static involved and essentially fried the card(s)?

 

[chinesestunna]

I'm siding with @kapone on this, are both cards showing same behavior? It sounds like cards might have issues if you've tried restoring to original config and same issues come up

 

[LunchB0xK1ller]

Any chance there was static involved and essentially fried the card(s)?

Well last night I booted the system up to try to get the information requested earlier and noticed that the cards showed up again so I decided to boot to dos and update the firmware. Well when I attempted to flash the first card the sas2flsh utility said the bios was corrupted but every flash I did failed. I emailed LSI and the responded today because there is something that is missing from my card programming that wont allow me to flash the bios on it. It failing because of a NVDATA mismatch. They are supposed to send me a image of a good card to try to flash that so the mismatch goes away. I am doubting static thought becasue even when the card isnt seen by the bios or OS the sas2flsh utility will see it and give me the warning about the corrupt bios.

All of this is only on one card though i still need to get the second card put back in a system and try to get the information requested through a ssh session into the linux side

 

[kapone]

That was my point. The BIOS and/or the NVRAM on the cards has somehow got corrupted, which can easily happen due to static (or other reasons). That's why the cards are acting flaky.

 

[LunchB0xK1ller]

That was my point. The BIOS and/or the NVRAM on the cards has somehow got corrupted, which can easily happen due to static (or other reasons). That's why the cards are acting flaky.

That makes complete sense then if static can do that. I will try the stuff that LSI sent to me to load on the card. I will also try the suggestions above with the second card.