RES3TV360 bricked? please help

Discussion in 'RAID Controllers and Host Bus Adapters' started by BusterJoe, Jul 29, 2018.

  1. BusterJoe

    BusterJoe New Member

    Joined:
    Jul 29, 2018
    Messages:
    8
    Likes Received:
    1
    I tried to do a firmware update to an Intel RES3TV360 which was connected to a Lenovo 930-16i (MegaRaid 9460-16i), following by mistake the steps for the IT/IR RAID Instead of using storcli:

    Code:
    Instructions for expander behind IT/IR RAID module or controller adapter
        (RS3FC044, RS3UC080, RS3UC080J, RS3GC008, RMS3JC080, RMS3VC160,
        RMSP3JD160J, RSP3QD160J, RSP3GD016J)
        NOTE: *** LINUX SYSTEM MUST HAVE A C COMPILER INSTALLED ***
    
        1. Download and extract all files to a USB key.
        2. Logon THE SYSTEM as root
        3. Connect USB key to system and mount it under Linux.
        4. Navigate to directory with the Linux utility.
        5. Install sg_utils.
            a. Navigate to USB drive and directory with the Linux utility.
            b. run "rpm -ivh sg3_utils-1.34-1.src.rpm"
                       This will install the sg_utils, creating a sg3_utils-1.34.tgz file in the "/root/rpmbuild/SOURCES" directory
            c. Navigate to the /root/rpmbuild/SOURCES directory
            d. Uncompress the sg3_utils-1.34.tgz (tar -xvf sg3_utils-1.34.tgz)
            e. Navigate to the /root/rpmbuild/SOURCES/sg3_utils-1.34 directory
            f. run ./configure
            g. run make
            h. run make install
        6. Install Lib Utils
            a. Navigate to USB drive and directory with the Linux utility.
            b. Install the Lib_Utils RPM, (rpm -ivh Lib_Utils-1.00-09.noarch.rpm)
        7. Find expander(s) device name.
            a. Run sg_scan -i
            b. The format of the expander device name is shown as below
               /dev/sgx
        8. Update expander fw.
            a. Navigate to USB drive and directory with the Linux utility.
            b. Run "./fw_tool.sh --fw_update ../../binary/firmware_image.bin <device name>"
    
    As result, two things went bad:

    1. The Intel RES3TV360 seem to be unresponsive: MegaRaid is not detecting the raid expander and the green LED on the RES3TV360 is constantly on. Any way to unbrick it? Maybe via JTAG?

    2. The MegaRaid is not being detected by the BIOS, however, in ESXi the drivers and storcli are able to see the card properly and Virtual Drivers are visible by the VMs. I’m guessing the MR EFI BIOS is corrupt, any thoughts on how to fix this?

    Thanks!
     
    #1
  2. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    570
    Likes Received:
    407
    the only thing you can do is cold power the system (completely remove power, then power it back on) to re-initialize the expander and hope it can start. It's also possible the only thing bad/screwed up is the megaraid card (which is possible considering you say it now doesn't show up in bios).

    reflash the latest firmware to your 9460-16i adapter, and see if it shows back up in bios - if it does, then try the correct way of updating the expander and see if the update tool/MR can now see the expander

    if none of that works, I would imagine you're probably out of luck - I don't see a JTAG or CPLD JTAG header anywhere on the card, and even if it did I would need to know exactly what embedded processor it's using and what type of flash etc
     
    #2
  3. BusterJoe

    BusterJoe New Member

    Joined:
    Jul 29, 2018
    Messages:
    8
    Likes Received:
    1
    I tried: disconnecting the power, BIOS reset, reflash MR to the latest firmware and even changing the raid controller PCIe slot, but none of that seem to help.

    There are a few flash chips on the SAS expander that I was able to identify, and my guess is that the firmware resides on the Micon N25Q128A SPI flash. I happened to have a cheap ISP programmer that seem to support it, but I've never done an in-circuit programming before, but it could be my last resort.

    As to the 9460-16i, does anyone know where can I get a BIOS and EFI BIOS file?
     
    #3
  4. mobilenvidia

    mobilenvidia Moderator

    Joined:
    Sep 25, 2011
    Messages:
    1,745
    Likes Received:
    37
    MegaRAID 9460-8i
     
    #4
  5. BusterJoe

    BusterJoe New Member

    Joined:
    Jul 29, 2018
    Messages:
    8
    Likes Received:
    1
    Thanks, I've already known about Broadcom's support download page, but I assumed that the BIOS can be obtained separately from the firmware, based on the storcli command line options and information that I found on MR recovery methods for other controllers.

    Anyway, I ended up downgrading the raid controller to a fairly old firmware, and now it is available in the BIOS, as long as the hard drives which were present during the firmware download are connected. If anyone of the disk groups is disconnected the controller won't be available in the BIOS.

    Is anyone able to explain this strange behavior?

    As to the SAS controller I was able to dump the Micon N25Q128A chip content using a programmer, and it seems to me that the firmware on it corrupted, e.g. PMC_BOOT section (partition?) is mostly blank while in other firmwares that I downloaded Intel there is some code.

    If I can get a dump from a working expander it would make my life a whole lot easier, but that would probably be too much to ask.

    So instead I believe that I would need to re-contract a proper image (128Mb => 16MB) in order to write it back, but this is very difficult to do without having the original firmware available, and on Intel site I can only find versions B024 or B057.

    Does anyone know where I can find firmware version B012 for RES3TV360/RES3FV288 or anything older then B024?

    Thanks
     
    #5
  6. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    570
    Likes Received:
    407
    if you can post or somehow send me the BIN/dump you made of the onboard flash, I can probably dig the correct section out of the latest firmware update package they have - version doesn't matter. You can then use your programmer to dump that back to the flash, and have a working controller. This is basically exactly what I had to do to publish the Delta 7024 > Dell 8024 crossflash

    since you're completely overwriting the entire flash, you don't need any certain version, you're completely replacing whatever was there before. You can just use the latest intel firmware image, and assuming a successful write, it'll boot up as a fully updated expander
     
    #6
    Tha_14 likes this.
  7. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    570
    Likes Received:
    407
    This is the image you need to write back to flash, the question is, starting from what offset - if you send me the dump of the existing flash I can give you an address. Likely it just starts right from 0 (top of flash)

    as for the megaraid bios, the notes from the latest firmware zip state that it's built into the bin image (versions listed below), I would image you just flash the latest firmware image and it takes care of the boot block as well according to the notes. There may be some flash command or argument to flash the BIOS section or leave it blank, not sure (I would check the manual):

    Code:
    Current Package Details:
    Firmware Package: 50.6.0-1375 (MR 7.6)
    Firmware 5.060.00-1455
    MR PL 07.00.00.00
    ROMENV 1.12
    BootBlock 7.02.00.00-0017
    NVDATA  5.0600.01-0008
    UEFI_Driver  0x07060500 (SIGNED)
    Hii 07.06.08.00 (SIGNED)
    FCODE 4.17.08.00
    BIOS 7.06.02.0
    Also, what flash/SPI programmer are you using?
     

    Attached Files:

    #7
  8. BusterJoe

    BusterJoe New Member

    Joined:
    Jul 29, 2018
    Messages:
    8
    Likes Received:
    1
    Attached the expander dump.

    Finding the offset is what I'm having trouble with, while trying to avoid writing to the flash too many times.

    I believe that the PMC_BOOT string (offset 0x17FEA) is a good starting point.

    But, I have some other concerns:

    In both B024 and B057 from the web there are 5 different "PMC_" strings, while in the dump there are 9, the extra 4 seem to be duplicates:

    PMC_BOOT
    PMC_ISTR x2
    PMC_NDSR x2
    PMC_LOG x2
    PMC_030608 x2


    I can see that both occurrences of PMC_ISTR section (up to to PMC_NDSR) are duplicates. And the same for both occurrences of PMC_NDSR (to PMC_LOG). And none of them match the ones from B057 that I tried to flash before.

    My guess would be that either it was reflashed twice in the past with the same version (the card is supposed to be new), or this is part of the corruption (bad rewrite recovery attempt after a failed write?)

    Also, PMC_BOOT sections in both B024 and B057 are the same size (97308 bytes) with very little difference. while in the dump its 4 bytes smaller while the content seems completely different and mostly blank. Which makes me believe that it is corrupt.

    The reason that I was looking for a B012 firmware was to confirm that the one in the dump is indeed messed up, and to help me figure out the original PMC_BOOT offset, in case that there is still references to it (not sure how the SPI flash exactly works). On my own I can deduct that the original PMC_BOOT offset is probably the last PMC_ISTR offset - 97304 bytes or 97308 bytes.

    I believe that this would leave me with three possible offsets.

    Please let me know what you think.

    Also, should I "blank" out the existing data before or after depending on which offset I go with?

    The programmer I'm using is an RT809F, with an SOIC16 clip working directly on the board.
     

    Attached Files:

    #8
  9. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    225
    Likes Received:
    37
    @BusterJoe The BIOS will only be loaded if there's disks attached to the controller. If there's no disks, there's no reason for the BIOS to remain resident as it's job is (primarily) to boot a disk.
     
    #9
  10. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    570
    Likes Received:
    407
    So looking through that dump, it seems the leading PMCBOOT string was a red herring. At the very beginning of flash (in your dump) is the boot code and config registers used to boot and set up the MIPS processor on the expander - which is good because this SHOULD be at the very beginning of flash.

    If you look at the firmware image from intel, it has that boot code and config setup bytes, but they don't start until about 30 bytes into the image, do a search in the firmware bin for "00 60 09 40 08 00 08" - this is where the actual firmware image to be flashed starts. Everything before that (including the PMC BOOT string) is just metadata/headers that tells the flashing utility what kind of image it is - but they get stripped away when written to flash.

    I went ahead and ripped these headers out, so the attached file starts with the proper boot code from address 0 as it should, so this simply needs to be written to flash starting from the very beginning (offset of 0). I would also completely erase the flash chip first, like you said it seems to have some backup partitions which are just copies of the original image (which is common to do when your flash chip is so much larger than the active firmware) but these copies also look corrupted. On successful boot it should copy the good code to the backup partitions if they are empty

    And don't worry about wearing out the flash, this Micron chip is rated for 100,000 erase/write cycles per sector (which is pretty typical for onboard flash).

    To summarize: erase the entire chip, then flash the attached bin file starting from address 0 (the very beginning of flash). Depending on your flashing software it may not want to flash this bin because the size isn't perfectly divisible by the sector size of the flash - if that happens let me know and I can pad it out. However most flashing software handles this automatically.

    After flashing, read the chip back (the first 1029060 bytes) and diff it with the attached firmware bin, they should be identical - this ensures the flash actually worked correctly. Then, fingers crossed, it should boot

    sidenote: when diffing the attached no-header firmware bin with your flash dump, your flash actually does still have the beginning couple hundred bytes of bootloader code unmodified, but you can see when it gets a few hundred bytes in, the firmware bin has all the OS code starting, but in your dump that whole area is completely blank - i would imagine that would be why it's not being recognized by the system :p
     

    Attached Files:

    #10
    Last edited: Aug 1, 2018
    leebo_28 likes this.
  11. BusterJoe

    BusterJoe New Member

    Joined:
    Jul 29, 2018
    Messages:
    8
    Likes Received:
    1
    Thanks, I flashed the attached file, I was able to confirm that the first 1029060 byte are identical (see attached), but it doesn't seem to work. Any other ideas?
     

    Attached Files:

    #11
  12. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    570
    Likes Received:
    407
    Well that's...strange. That means there's content that needs to be in flash that intel doesn't include in their flash/firmware package, which is quite bad practice (not something I expect from intel)

    There's a chance whatever it needed was still in your flash, try flashing back your original "bad flash dump" (not the new one after you erased the whole tihng), then dump the no-header firmware bin I posted above on top of that to fix the corrupted bootloader section

    if that still doesn't work, do you have another sas controller/HBA you can connect this card to? I would begin to expect maybe it's this strangely acting megaraid controller that could be causing issues
     
    #12
    Last edited: Aug 1, 2018
    BusterJoe likes this.
  13. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    570
    Likes Received:
    407
    Last idea for now, try flashing this from offset 0 and see if it works - it's the unmodified (headers included) latest FW image. it's possible it DOES need that small header code and extra bytes at the beginning, and they were simply missing from your original dump because it got flashed wrong
     

    Attached Files:

    #13
  14. BusterJoe

    BusterJoe New Member

    Joined:
    Jul 29, 2018
    Messages:
    8
    Likes Received:
    1
    Yeah, I'm not sure that the raid controller is working 100% as expected, and unfortunately I don't have another sas controller to test with.

    Is this for sure an expected behavior from a MegaRaid?

    When I initially got the card I tested it without any hard drives connected and it was visible in the BIOS, but there was no configuration set at the time.

    However, if I disconnect one the disks belonging to one of the three virtual drives that I've setup (meaning there are still some disks connected), the controller will not show up in the BIOS. So, seems a bit strange.

    Sure I'll give it a try, thanks.
     
    #14
  15. BusterJoe

    BusterJoe New Member

    Joined:
    Jul 29, 2018
    Messages:
    8
    Likes Received:
    1
    Unfortunately, flashing the B057 firmware (last one you uploaded) as is doesn't work.

    It's getting here pretty late, so I'll try your suggestion with the merging the original dump with the no headers firmware tomorrow.

    I might also take out my logic analyzer and see if the controller is sending useful debug information on any of the pins.
     
    #15
  16. BusterJoe

    BusterJoe New Member

    Joined:
    Jul 29, 2018
    Messages:
    8
    Likes Received:
    1
    Good news, this did the trick, and my expander is working back again.

    The programmer software that I was using had an auto erase feature before every write. I couldn’t find a way to disable it, so I ended up manually merging both the corrupted firmware dump with the "no headers" firmware and writing it back to the flash, which then it was able to boot.

    BTW, the expander was showing that it was still in version B012. This makes me believe that after loading the initial boot code from the beginning of the flash it skips to a different offset to load the rest of the firmware, which was in the old dump.

    Then using storcli I upgraded the expander firmware to B057 and generated another dump of the firmware to compare with. What I found interesting was that the upgraded flash was modified in three different locations. So I’m guessing that when upgrading via the expander in normal mode, there is more work being done in the background such as managing the firmware location on the flash and setting offset registers.

    For reference, in case anyone else comes across a similar issue, I’ve uploaded both the merged flash dump (B012) that brought my controller back to life, and a working dump of the flash after the B057 upgrade.

    Tip: From my observation, when my expander was bricked, the green LED was constantly on. And when the controller is in normal working condition the LED starts blinking slowly on and off (at 1 second interval) after the power is turned on for a few seconds, and it doesn't need to be connected to an HBA or RAID controller. This could be a quick way to test that the new firmware was successful after it was uploaded to the chip.

    Disclaimer: The uploaded flash dumps are far from an ideal solution and may potentially contain corrupt data that could result in unexpected behavior or completely bricking your SAS expander. My recommendation is to only use them if you know what you are doing and as a last resort, where no better option is available. And make sure to backup the firmware from your own controller first. I take no responsibility if it damages your SAS expander.


    @fohdeesha, thank you for your time and help, you’ve been a life saver!
     

    Attached Files:

    #16
    fohdeesha likes this.
  17. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    570
    Likes Received:
    407
    Nice! glad to hear it. good thinking uploading a final working dump, if anyone ever runs into this hopefully they'll find this post via google. If you haven't I'd update the CPLD firmware on it too, I think it's included in the same ZIP - I remember seeing a release note saying these should match. granted in my experience with CPLD's, all they do in these types of components is control LED's
     
    #17
Similar Threads: RES3TV360 bricked
Forum Title Date
RAID Controllers and Host Bus Adapters Dell H330 with RES3TV360 - not working Dec 23, 2017
RAID Controllers and Host Bus Adapters Intel 36 port expander usage issue Pls Help! ( Intel RES3tv360 expander) Sep 10, 2016
RAID Controllers and Host Bus Adapters Intel RES3TV360 Jan 26, 2016
RAID Controllers and Host Bus Adapters Benchmrks! : Adaptec 8405 -> RES3TV360 Expander + benchmarks/info May 3, 2015
RAID Controllers and Host Bus Adapters bricked intel AXXRMS2AF040 Feb 3, 2018

Share This Page