Fusion-io ioDrive 2 1.2TB Reference Page

gb00s · Feb 23, 2022

If I remember correctly it should be like:

Code:

fio-update-iodrive /folder-of-firmware/xyz_*.fff

Of course, the drive has to be detached before the upgrade via

Code:

fio-detach /dev/fctXYZ

You can also test (pretend the upgrade) for any error messages before the upgrade via

Code:

fio-update-iodrive -p /folder-of-firmware/xyz_*.fff

miha_ · Feb 23, 2022

Did try and get the following error :

C:\WINDOWS\system32>fio-update-iodrive -p C:\SanDisk\fusion_3.2.11-20150618.fff
WARNING: DO NOT TURN OFF POWER OR RUN ANY IODRIVE UTILITIES WHILE THE FIRMWARE UPDATE IS IN PROGRESS
Please wait...this could take a while

PRETENDING: no changes are being made.

Updating: [====================] (100%)
Error: Firmware file does not contain firmware for device 'fct1', part 'PA004151015'.

gb00s · Feb 23, 2022

can you do a

Code:

fio-update-iodrive -l C:\SanDisk\fusion_3.2.11-20150618.fff

and show us the output?

miha_ · Feb 23, 2022

I did attached it

gb00s · Feb 23, 2022

If I'm not mistaken there's no driver for your device 'PA004151015' in the fff-file you are trying to use.

miha_ · Feb 23, 2022

Yes, I was scratching my head with this also, as the card works fine with the drivers also performance is within specs. So I did gues I'm not the only person who has this problem and did ask

acquacow · Feb 23, 2022

That's likely an HP OEM'd ioDrive. You have to create entries for it in the firmware file you download from SanDisk so that it can be current.

This is

[PA004154001]
version = 7.1.17.116786
file = gen2_49_salmon_hp_mono_1fp_7.1.17.116786.bin
format = bin
ecc = 49b
cntrpdiversion = 1.0.35
cntrpdifile = carre_1.0.35.pdi
cntrpdiformat = pdi
optrom = 1e00000
optromversion = 3.2.11.1581
optromfile = uefi-3.2.11.1581.rom

Above is the most similar entry from the HP INFO file. You'd want to add that heading to

You should be able to drop the following into the INFO file from the latest sandisk, zip it back up, rename it back to .fff and flash with it:

[ PA004151015 ]
version = 7.1.17.11678
file = gen2_49_salmon_fusion_7.1.17.116786.bin
format = bin
ecc = 49b
cntrpdiversion = 1.0.35
cntrpdifile = carre_1.0.35.pdi
cntrpdiformat = pdi
optrom = 1e00000
optromversion = 3.2.16.1731
optromfile = uefi-3.2.16.1731.rom

I would like to find an INFO file that has the proper prefix for your card to be sure, but this is just some rough fudging that should work.

No guarantees though,

-- Dave

gb00s · Feb 24, 2022

acquacow said:
That's likely an HP OEM'd ioDrive. You have to create entries for it in the firmware file you download from SanDisk so that it can be current.

This is

[PA004154001]
version = 7.1.17.116786
file = gen2_49_salmon_hp_mono_1fp_7.1.17.116786.bin
format = bin
ecc = 49b
cntrpdiversion = 1.0.35
cntrpdifile = carre_1.0.35.pdi
cntrpdiformat = pdi
optrom = 1e00000
optromversion = 3.2.11.1581
optromfile = uefi-3.2.11.1581.rom

Above is the most similar entry from the HP INFO file. You'd want to add that heading to

You should be able to drop the following into the INFO file from the latest sandisk, zip it back up, rename it back to .fff and flash with it:

[ PA004151015 ]
version = 7.1.17.11678
file = gen2_49_salmon_fusion_7.1.17.116786.bin
format = bin
ecc = 49b
cntrpdiversion = 1.0.35
cntrpdifile = carre_1.0.35.pdi
cntrpdiformat = pdi
optrom = 1e00000
optromversion = 3.2.16.1731
optromfile = uefi-3.2.16.1731.rom

I would like to find an INFO file that has the proper prefix for your card to be sure, but this is just some rough fudging that should work.

No guarantees though,

-- Dave

Nice idea Dave, my success rate for this solution is 50/50

miha_ · Feb 24, 2022

did roll the dice and update was successful. Fist observations is that it performs a bit better and faster.

l1wendell · Feb 24, 2022

I have a fusion iodrive2 [ SanDisk ioDrive2 (rev 04)] that was being used in windows but windows did a BSOD and now I can't get it to attach:

https://twitter.com/i/web/status/1496732811713204224

I swiched to Linux for experimentation and debugging.

I also had some "test" iodrive2 drives that I could experiment on and so I was able to use those to try various things like sweeping from firmware versions from 3.2.2 o 3.2.6 to 3.2.8 to 3.2.11 and 3.2.16 (it originall had 3.2.11 which was bundled with the windows driver in use). None of that seems to allow the drive to recover when it is in this "read error" state.

When running fio-attach on linux I get the message about unclean shudown shown above and it seems to be doing the same as fio-attach -r (it takes a few minutes). At 53% it errors out with a read error on the drive I need to recover from.

If I could mount the drive read only or even just put it in some kind of mode to dump the raw nand I'd be happy.

This had reFS on it under windows, but the backups of the refs volume were also corrupt and that is a "ReFS thing" so I'm kind of stuck.

I tried setting disable_scanner to 1 but that also prevents the kernel module from reading some of the low-level start up information I suppose from the leb map and and so fio-attach will no work after that.

Running strings on the .ko kernel module and fio-attach it "seems like" there is some code in there for attaching read only but...? Not much documentation.

I tried some parameters from here:

iomemory-vsl/module_param.c at master · sjuxax/iomemory-vsl

A version of iomemory-vsl that compiles on Ubuntu 15.10, kernel 4.2. Probably not safe to use. - sjuxax/iomemory-vsl

github.com

when inserting the kernel module on the "test" fusion io drives and nothing seemed to make a difference about skipping a rescan when an unclean shutdown was detected or attaching/mounting read only. On the test devices I loaded a bunch of data before doing experiments and firmware version sweeps and am happy to report no data was lost on the test devices.

I may end up sending this to a lab to attempt recovery or to dump the nand. Help me @acquacow you're my only hope? lol

acquacow · Feb 26, 2022

Hmm, I'll have to see if I can poke around at some of mine and see if I can get them to load.

In all my years at Fusion-io/SanDisk, I've never seen that error.

I have had ReFS lose all my data multiple times on multiple devices though, so I just don't touch it anymore, I stick to ZFS.

Dumping the nand will be quite difficult, as the data is laid out randomly on the physical device and only exists in block format at the driver layer once assembled. There's nothing low-level that resembles a filesystem/data.

Downgrading firmware versions is also not a good idea, as the data structures/etc have changed over time and going backwards is a good way to corrupt all kinds of card metadata.

I'll dig through my old archive of stuff tonight and see if I can find anything useful.

-- Dave

PS: Love your stuff, can't wait for all the openshift updates since I'm now doing a lot of openshift stuff at RH.

AMiskell · Apr 9, 2022

@acquacow Do you know if there's a way to fix a SX350 card that's reporting a missing LEB map?

I got a couple of these cards and one won't attach due to it reporting a missing LEB map and there's not much information I've been able to find on how to fix that.

Code:

[   19.448558] fioinf ioMemory SX350-3200 0000:01:00.0: Failed to read either copy of the MEL.
[   19.448560] fioinf ioMemory SX350-3200 0000:01:00.0: Unable to read management data - MEL.
[   19.448562] fiowrn ioMemory SX350-3200 0000:01:00.0: Unable to read management data - Attach disabled (-5).
[   19.448683] fioinf ioMemory SX350-3200 0000:01:00.0: No lebmap available, recovering data may take some time.
[   21.196319] fioinf ioMemory SX350-3200 0000:01:00.0: NV data format version.
[   21.196322] fioinf ioMemory SX350-3200 0000:01:00.0: Unable to recover management NV data (-2).
[   21.196323] fioinf ioMemory SX350-3200 0000:01:00.0: Init NV data
[   21.196390] fioerr ioMemory SX350-3200 0000:01:00.0: 18456 Unable to attach - explicitly disabled.
[   21.196584] fioinf ioMemory SX350-3200 0000:01:00.0: probed fct0
[   21.197592] fioerr ioMemory SX350-3200 0000:01:00.0: 18461 Auto attach failed with error EINVAL: Invalid argument

root@ubuntu:~# fio-status

Found 1 VSL driver package:
   4.3.7 build 1205 Driver: loaded

Found 1 ioMemory device in this system

Adapter: ioMono  (driver 4.3.7)
    ioMemory SX350-3200, Product Number:HJ7HN, SN:US0HJ7HNSNS0075U0084A00
    PCIe Power limit threshold: 24.75W
    Connected ioMemory modules:
      fct0: 01:00.0,     Product Number:HJ7HN, SN:US0HJ7HNSNS0075U0084A00

fct0    Detached
    ioMemory Adapter Controller, Product Number:HJ7HN, SN:1721D00C4
!! --->    There are active errors or warnings on this device!  Read below for details.
    PCI:01:00.0, Slot Number:1
    Firmware v8.9.9, rev 20200113 Public
    15816428544.00 GBytes device size
    Internal temperature: 59.55 degC, max 64.97 degC
    Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%

    ACTIVE WARNINGS:
        The bandwidth of the PCI slot is not optimal for the ioMemory.
        The ioMemory is missing a LEB map.

acquacow · Apr 9, 2022

You seem to have quite a few possible issues there.

Have you tried to sure-erase and fio-format the card yet?

AMiskell · Apr 9, 2022

acquacow said:
You seem to have quite a few possible issues there.

Have you tried to sure-erase and fio-format the card yet?

Yes, I have neither worked in my case.

Code:

root@ubuntu:~# fio-format /dev/fct0

/dev/fct0: Error: Device does not have a persistent LEB map.


root@ubuntu:~# fio-sure-erase /dev/fct0
NOTE: Default options are not NIST 800-88 compilant.
WARNING: Default options may degrade your card prematurely.
         Use --nist for better longevity.
Do you wish to continue [y/n]? y

WARNING: sanitizing will destroy any existing data on the device!
Do you wish to continue [y/n]? y
Erasing blocks: [====================] (100%)
Error: Device '/dev/fct0' had an error while sanitizing. Could not clear flash data. Make sure the device is detached (by running fio-detach) and not running another operation.
       To ensure data erasure, re-run with the --nist or --purge parameter,
         which will erase all user data but may make the card inoperable
Failure: ioMemory SX350-3200 PN:SDFADAMOS-3T20-SD1 SN:1721D00C4 FIO-SN:1721D00C4 Device:/dev/fct0

AMiskell · Apr 9, 2022

AMiskell said:

Yes, I have neither worked in my case.

Code:

root@ubuntu:~# fio-format /dev/fct0

/dev/fct0: Error: Device does not have a persistent LEB map.


root@ubuntu:~# fio-sure-erase /dev/fct0
NOTE: Default options are not NIST 800-88 compilant.
WARNING: Default options may degrade your card prematurely.
         Use --nist for better longevity.
Do you wish to continue [y/n]? y

WARNING: sanitizing will destroy any existing data on the device!
Do you wish to continue [y/n]? y
Erasing blocks: [====================] (100%)
Error: Device '/dev/fct0' had an error while sanitizing. Could not clear flash data. Make sure the device is detached (by running fio-detach) and not running another operation.
       To ensure data erasure, re-run with the --nist or --purge parameter,
         which will erase all user data but may make the card inoperable
Failure: ioMemory SX350-3200 PN:SDFADAMOS-3T20-SD1 SN:1721D00C4 FIO-SN:1721D00C4 Device:/dev/fct0

I finally got fio-secure-erase to work. I had to use the -t option to reset to factory defaults. Now the LEB map messages are gone. However, now it won't let me format the drive.

Below is the latest messages at boot and the output of fio-format.

Code:

[   18.909766] fioinf ioDrive 0000:01:00.0: Starting device ioMemory SX350-3200 0000:01:00.0
[   88.347128] fioinf ioMemory SX350-3200 0000:01:00.0: Unable to recover management NV data (-2).
[   88.347130] fioinf ioMemory SX350-3200 0000:01:00.0: Init NV data
[   88.347282] fioinf ioMemory SX350-3200 0000:01:00.0: probed fct0
[   88.348887] fioinf ioMemory SX350-3200 0000:01:00.0: ***************************************************
[   88.348889] fioinf ioMemory SX350-3200 0000:01:00.0: ***            No media format found!           ***
[   88.348889] fioinf ioMemory SX350-3200 0000:01:00.0: ***      Please contact Customer Support.       ***
[   88.348890] fioinf ioMemory SX350-3200 0000:01:00.0: ***************************************************
[   88.348922] fioerr ioMemory SX350-3200 0000:01:00.0: 18433 Unable to attach - ebo (-2).
[   88.350397] fioerr ioMemory SX350-3200 0000:01:00.0: 18455 Unable to attach device (-2).
[   88.351579] fioerr ioMemory SX350-3200 0000:01:00.0: 18460 Unable to attach - failed to create device: No such file or directory (-2).
[   88.352778] fioerr ioMemory SX350-3200 0000:01:00.0: 18461 Auto attach failed with error ENOENT: No such file or directory


root@ubuntu:~# fio-format /dev/fct0

/dev/fct0: Creating block device.
  Block device of size 3200.00GBytes (2980.23GiBytes).
  Using default block (sector) size of 4096 bytes.

WARNING: Formatting will destroy any existing data on the device!
Do you wish to continue [y/n]? y

WARNING: Do not interrupt formatting!

Formatting: [====================] (100%)

Error: Device '/dev/fct0' had an error while formatting: Invalid input.

Additional details about the error may be found in system logs.
Please see documentation or contact support for further assistance.

gb00s · Apr 9, 2022

This is a common issue that you can erase the 'card' but not format it with previous LEB-map-errors. I suspect hardware map (soft?) failure. I don't know anyone who could revive a card from LEB-map-errors.

acquacow · Apr 9, 2022

I've had to do multiple sure-erase/format cycles on cards sometimes to get them back into working order.

I found some leb map notes the other day, I'll have to see if they apply to the SX350s...

hedehodo · Apr 27, 2022

We have "HP 1.0TB HH/HL Light Endurance (LE) PCIe Workload Accelerator". This card only has driver up to RHEL7. I tried to search web and found that a modified VSL4.3.7 driver released. I installed fedora35(kernel 5.6) on server and also installed this modified driver. But dmesg said that I had to upgrade firmware on card. I found an firmware collection but it did not include my card's model, and add a line that includes my cards id and upgrade the firmware. Didn't worked. What else should I try?

acquacow · Apr 27, 2022

hedehodo said:
We have "HP 1.0TB HH/HL Light Endurance (LE) PCIe Workload Accelerator". This card only has driver up to RHEL7. I tried to search web and found that a modified VSL4.3.7 driver released. I installed fedora35(kernel 5.6) on server and also installed this modified driver. But dmesg said that I had to upgrade firmware on card. I found an firmware collection but it did not include my card's model, and add a line that includes my cards id and upgrade the firmware. Didn't worked. What else should I try?

What card do you specifically have? We never had a 1.0TB card in our line-up.

Looks like HP actually did release a 1.0TB version... I'm just not sure if that's an ioScale 825 overprovisioned to 1.0TB or a 1.6TB under provisioned to 1.0TB... hmm, my guess is the prior.

If it is an ioScale 825GB card, then it may actually be a VSL 3.x card and not 4.x...

Any photos?

Thanks,

-- Dave

hedehodo · Apr 28, 2022

acquacow said:
What card do you specifically have? We never had a 1.0TB card in our line-up.

Looks like HP actually did release a 1.0TB version... I'm just not sure if that's an ioScale 825 overprovisioned to 1.0TB or a 1.6TB under provisioned to 1.0TB... hmm, my guess is the prior.

If it is an ioScale 825GB card, then it may actually be a VSL 3.x card and not 4.x...

Any photos?

Thanks,

-- Dave

Here is the card;

HPE Support Center

support.hpe.com

Actually I solved the problem. I installed modified VSL4 and find newer firmware on HP site. Here is the firmware that I used.

HPE Support Center

support.hpe.com

Fusion-io ioDrive 2 1.2TB Reference Page

Well-Known Member

New Member

Well-Known Member

New Member

Attachments

Well-Known Member

New Member

Well-Known Member

Well-Known Member

New Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

New Member

Well-Known Member

Well-Known Member

New Member

Well-Known Member

New Member