IODrive2 - General Channel Initialization failure

Jason Antes

Active Member
Feb 28, 2020
211
72
28
Twin Cities
Wondering if my 1.2TB IODrive2 is salvageable. It recently stopped working and running an fio-check gave me:
Code:
Adapter: Single Controller Adapter
        Fusion-io ioDrive2 1.205TB, Product Number:F00-001-1T20-CS-0001, SN:1208D5248, FIO SN:1208D5248
        ioDrive2 Adapter Controller, PN:PA004137009
        External Power Override: ON
        External Power: NOT connected
        PCIe Power limit threshold: Disabled
        PCIe slot available power: unavailable
        Connected ioMemory modules:
          fct3: Product Number:F00-001-1T20-CS-0001, SN:1208D5248

fct3    Status unknown: Driver is in MINIMAL MODE:
                General channel initialization failure
        ioDrive2 Adapter Controller, Product Number:F00-001-1T20-CS-0001, SN:1208D5248
!! ---> There are active errors or warnings on this device!  Read below for details.
        ioDrive2 Adapter Controller, PN:PA004137009
        SMP(AVR) Versions: App Version: 1.0.35.0, Boot Version: 0.0.9.1
        Located in slot 0 Center of ioDrive2 Adapter Controller SN:1208D5248
        Powerloss protection: not available
        PCI:08:00.0, Slot Number:1
        Vendor:1aed, Device:2001, Sub vendor:1aed, Sub device:2001
        Firmware v7.1.17, rev 116786 Public
        Geometry and capacity information not available.
        Format: not low-level formatted
        PCIe slot available power: 26.00W
        PCIe negotiated link: 4 lanes at 5.0 Gt/sec each, 2000.00 MBytes/sec total
        Internal temperature: 63.49 degC, max 74.81 degC
        Internal voltage: avg 1.02V, max 1.02V
        Aux voltage: avg 2.48V, max 2.49V
        Rated PBW: 17.00 PB
        Lifetime data volumes:
           Physical bytes written: 0
           Physical bytes read   : 0
        RAM usage:
           Current: 0 bytes
           Peak   : 0 bytes

        ACTIVE WARNINGS:
            The ioMemory is currently running in a minimal state.
fio-pci-check didn't show any problems with the PCIE bus and I've been running 2 of these cards for several years without issue. Digging into the syslog I found these with relation to the problem:
Code:
2022-10-03T02:15:49.9641956Z - ERROR - FusionEventDriver - fct3: MINIMAL MODE DRIVER: General channel initialization failure. error code:-1
2022-10-03T02:15:49.4956171Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 0 byte 1 exp: 0xa8 act: 0x01
2022-10-03T02:15:49.4956231Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 0 byte 2 exp: 0x25 act: 0x02
2022-10-03T02:15:49.4956272Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 0 byte 3 exp: 0xcb act: 0x03
2022-10-03T02:15:49.4956314Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 1 byte 1 exp: 0xa8 act: 0x01
2022-10-03T02:15:49.4956353Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 1 byte 2 exp: 0x25 act: 0x02
2022-10-03T02:15:49.4956392Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 1 byte 3 exp: 0xcb act: 0x03
2022-10-03T02:15:49.4956432Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 2 byte 1 exp: 0xa8 act: 0x01
2022-10-03T02:15:49.4956471Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 2 byte 2 exp: 0x25 act: 0x02
2022-10-03T02:15:49.4956509Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 2 byte 3 exp: 0xcb act: 0x03
2022-10-03T02:15:49.4956550Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 3 byte 1 exp: 0xa8 act: 0x01
2022-10-03T02:15:49.4956588Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 3 byte 2 exp: 0x25 act: 0x02
2022-10-03T02:15:49.4956627Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 3 byte 3 exp: 0xcb act: 0x03
Code:
C:\WINDOWS\system32>fio-read-lebmap /dev/fct3
unexpected FUSION_IOCTL_QUERY_MEL result -22 (22).
Error: Failed to read MEL: Invalid input.
The system runs the consistency check on the still working drive without issue, but doesn't see this drive at all. Is it fubar or is there a way to remap those bad pads or force it online? Attach and detach don't work.

Thanks,
Jason
 

Jason Antes

Active Member
Feb 28, 2020
211
72
28
Twin Cities
This is the full initialization out of the syslog. Looks good until it gets to those pad errors (I think).
Code:
2022-10-03T02:10:10.5633369Z - INFO - FusionEventDriver - VSL configuration hash: 22e21ad8f72d48e3f25527ac40b81d79055f95b3
2022-10-03T02:10:10.5635738Z - INFO - FusionEventDriver - ioMemory VSL v 3.2.15.1699 pinnacles@f0f84521e1b1 Built on Apr 10 2017 11:58:09
2022-10-03T02:10:10.5636953Z - INFO - FusionEventDriver - fct3 at 0000:08:00.0
2022-10-03T02:10:10.5702572Z - INFO - FusionEventDriver - 0000:08:00.0: BIOS affinity configuration found - setting to node 0
2022-10-03T02:10:10.5799209Z - INFO - FusionEventDriver - 0000:08:00.0: requested interrupt affinity group 0 (node 0), group-relative processor mask 0x0000000000000001
2022-10-03T02:10:10.7044432Z - INFO - FusionEventDriver - fct3: Starting master controller
2022-10-03T02:10:10.7676418Z - INFO - FusionEventDriver - fct3: PMP Address: 1 1 1
2022-10-03T02:10:10.8302431Z - INFO - FusionEventDriver - fct3: SMP Controller Firmware APP version 1.0.35 0
2022-10-03T02:10:10.8302453Z - INFO - FusionEventDriver - fct3: SMP Controller Firmware BOOT version 0.0.9 1
2022-10-03T02:10:11.3353214Z - INFO - FusionEventDriver - fct3: Required PCIE bandwidth 2.000 GBytes per sec
2022-10-03T02:10:11.3353258Z - INFO - FusionEventDriver - fct3: Board serial number is 1215D0142
2022-10-03T02:10:11.3353280Z - INFO - FusionEventDriver - fct3: Adapter serial number is 1215D0142
2022-10-03T02:10:11.3353320Z - INFO - FusionEventDriver - fct3: Default capacity 1205.000 GBytes
2022-10-03T02:10:11.3353342Z - INFO - FusionEventDriver - fct3: Default sector size 512 bytes
2022-10-03T02:10:11.3353363Z - INFO - FusionEventDriver - fct3: Rated endurance 17.00 PBytes
2022-10-03T02:10:11.3353383Z - INFO - FusionEventDriver - fct3: 100C temp range hardware found
2022-10-03T02:10:11.8353514Z - INFO - FusionEventDriver - fct3: Adapter: external power override enabled
2022-10-03T02:10:11.8353737Z - INFO - FusionEventDriver - fct3: Firmware version 7.1.17 116786 (0x700411 0x1c832)
2022-10-03T02:10:11.8353758Z - INFO - FusionEventDriver - fct3: Platform version 16
2022-10-03T02:10:11.8353780Z - INFO - FusionEventDriver - fct3: Firmware VCS version 116786 [0x1c832]
2022-10-03T02:10:11.8353918Z - INFO - FusionEventDriver - fct3: Firmware VCS uid 0xaeb15671994a45642f91efbb214fa428e4245f8a
2022-10-03T02:10:11.8661738Z - INFO - FusionEventDriver - fct3: Powercut flush: Enabled
2022-10-03T02:10:12.4909218Z - INFO - FusionEventDriver - fct3: PCIe power monitor: disabled
2022-10-03T02:10:12.4909248Z - INFO - FusionEventDriver - fct3: Thermal monitoring: Enabled
2022-10-03T02:10:12.4909282Z - INFO - FusionEventDriver - fct3: Hardware temperature alarm set for 100C.
2022-10-03T02:10:14.3885575Z - INFO - FusionEventDriver - fct3-1215D0142: Attempting to load ioDrive.
2022-10-03T02:10:14.4788810Z - INFO - FusionEventDriver - fct3-1215D0142: sector_size=512
2022-10-03T02:10:14.4788848Z - INFO - FusionEventDriver - fct3-1215D0142: setting channel range data to [2 .. 2047]
2022-10-03T02:10:14.9263584Z - INFO - FusionEventDriver - fct3-1215D0142: Found metadata in EBs 1487-1487, loading...
2022-10-03T02:10:15.4670418Z - INFO - FusionEventDriver - fct3-1215D0142: setting recovered append point 1487+749518848
2022-10-03T02:10:15.4881064Z - INFO - FusionEventDriver - fct3-1215D0142: Creating device of size 1205000000000 bytes with 2353515625 sectors of 512 bytes (1412115131 mapped).
2022-10-03T02:10:15.4881093Z - INFO - FusionEventDriver - fct3-1215D0142: Creating block device internal sector size: 512...
2022-10-03T02:10:15.4881356Z - INFO - FusionEventDriver - fct3-1215D0142: Attach succeeded.
2022-10-03T02:10:15.4881377Z - INFO - FusionEventDriver - fct3-1215D0142: ioDrive was successfully loaded and attached to the system.
2022-10-03T02:10:15.4886458Z - INFO - FusionEventDriver - Creating block device external sector size: 512...
2022-10-03T02:10:21.5568395Z - INFO - FusionEventDriver - fiochkdrv: starting
2022-10-03T02:10:21.5568463Z - INFO - FusionEventDriver - fiochkdrv: no unattached drives found
2022-10-03T02:10:21.5568515Z - INFO - FusionEventDriver - fiochkdrv: stopping
2022-10-03T02:15:47.5774424Z - INFO - FusionEventDriver - VSL configuration hash: 22e21ad8f72d48e3f25527ac40b81d79055f95b3
2022-10-03T02:15:47.5777032Z - INFO - FusionEventDriver - ioMemory VSL v 3.2.15.1699 pinnacles@f0f84521e1b1 Built on Apr 10 2017 11:58:09
2022-10-03T02:15:47.5778300Z - INFO - FusionEventDriver - fct2 at 0000:0d:00.0
2022-10-03T02:15:47.5845023Z - INFO - FusionEventDriver - 0000:0d:00.0: BIOS affinity configuration found - setting to node 0
2022-10-03T02:15:47.5947208Z - INFO - FusionEventDriver - fct3 at 0000:08:00.0
2022-10-03T02:15:47.6013676Z - INFO - FusionEventDriver - 0000:08:00.0: BIOS affinity configuration found - setting to node 0
2022-10-03T02:15:47.6015019Z - INFO - FusionEventDriver - 0000:08:00.0: requested interrupt affinity group 0 (node 0), group-relative processor mask 0x0000000000000002
2022-10-03T02:15:47.6015920Z - INFO - FusionEventDriver - 0000:0d:00.0: requested interrupt affinity group 0 (node 0), group-relative processor mask 0x0000000000000001
2022-10-03T02:15:47.7362582Z - INFO - FusionEventDriver - fct2: Starting master controller
2022-10-03T02:15:47.7362588Z - INFO - FusionEventDriver - fct3: Starting master controller
2022-10-03T02:15:47.7995169Z - INFO - FusionEventDriver - fct3: PMP Address: 1 1 1
2022-10-03T02:15:47.8306945Z - INFO - FusionEventDriver - fct2: PMP Address: 1 1 1
2022-10-03T02:15:47.8620762Z - INFO - FusionEventDriver - fct3: SMP Controller Firmware APP version 1.0.35 0
2022-10-03T02:15:47.8620793Z - INFO - FusionEventDriver - fct3: SMP Controller Firmware BOOT version 0.0.9 1
2022-10-03T02:15:48.0575772Z - INFO - FusionEventDriver - fct2: SMP Controller Firmware APP version 1.0.35 0
2022-10-03T02:15:48.0575813Z - INFO - FusionEventDriver - fct2: SMP Controller Firmware BOOT version 0.0.9 1
2022-10-03T02:15:48.4015791Z - INFO - FusionEventDriver - fct3: Required PCIE bandwidth 2.000 GBytes per sec
2022-10-03T02:15:48.4015830Z - INFO - FusionEventDriver - fct3: Board serial number is 1208D5248
2022-10-03T02:15:48.4015851Z - INFO - FusionEventDriver - fct3: Adapter serial number is 1208D5248
2022-10-03T02:15:48.4015892Z - INFO - FusionEventDriver - fct3: Default capacity 1205.000 GBytes
2022-10-03T02:15:48.4015913Z - INFO - FusionEventDriver - fct3: Default sector size 512 bytes
2022-10-03T02:15:48.4015935Z - INFO - FusionEventDriver - fct3: Rated endurance 17.00 PBytes
2022-10-03T02:15:48.4015954Z - INFO - FusionEventDriver - fct3: 100C temp range hardware found
2022-10-03T02:15:48.5890796Z - INFO - FusionEventDriver - fct2: Required PCIE bandwidth 2.000 GBytes per sec
2022-10-03T02:15:48.5890837Z - INFO - FusionEventDriver - fct2: Board serial number is 1215D0142
2022-10-03T02:15:48.5890857Z - INFO - FusionEventDriver - fct2: Adapter serial number is 1215D0142
2022-10-03T02:15:48.5890902Z - INFO - FusionEventDriver - fct2: Default capacity 1205.000 GBytes
2022-10-03T02:15:48.5890924Z - INFO - FusionEventDriver - fct2: Default sector size 512 bytes
2022-10-03T02:15:48.5890945Z - INFO - FusionEventDriver - fct2: Rated endurance 17.00 PBytes
2022-10-03T02:15:48.5890964Z - INFO - FusionEventDriver - fct2: 100C temp range hardware found
2022-10-03T02:15:48.9017192Z - INFO - FusionEventDriver - fct3: Adapter: external power override enabled
2022-10-03T02:15:48.9017412Z - INFO - FusionEventDriver - fct3: Firmware version 7.1.17 116786 (0x700411 0x1c832)
2022-10-03T02:15:48.9017434Z - INFO - FusionEventDriver - fct3: Platform version 16
2022-10-03T02:15:48.9017456Z - INFO - FusionEventDriver - fct3: Firmware VCS version 116786 [0x1c832]
2022-10-03T02:15:48.9017595Z - INFO - FusionEventDriver - fct3: Firmware VCS uid 0xaeb15671994a45642f91efbb214fa428e4245f8a
2022-10-03T02:15:48.9325449Z - INFO - FusionEventDriver - fct3: Powercut flush: Enabled
2022-10-03T02:15:49.0891084Z - INFO - FusionEventDriver - fct2: Adapter: external power override enabled
2022-10-03T02:15:49.0891253Z - INFO - FusionEventDriver - fct2: Firmware version 7.1.17 116786 (0x700411 0x1c832)
2022-10-03T02:15:49.0891274Z - INFO - FusionEventDriver - fct2: Platform version 16
2022-10-03T02:15:49.0891295Z - INFO - FusionEventDriver - fct2: Firmware VCS version 116786 [0x1c832]
2022-10-03T02:15:49.0891401Z - INFO - FusionEventDriver - fct2: Firmware VCS uid 0xaeb15671994a45642f91efbb214fa428e4245f8a
2022-10-03T02:15:49.1200601Z - INFO - FusionEventDriver - fct2: Powercut flush: Enabled
2022-10-03T02:15:49.4956171Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 0 byte 1 exp: 0xa8 act: 0x01
2022-10-03T02:15:49.4956231Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 0 byte 2 exp: 0x25 act: 0x02
2022-10-03T02:15:49.4956272Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 0 byte 3 exp: 0xcb act: 0x03
2022-10-03T02:15:49.4956314Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 1 byte 1 exp: 0xa8 act: 0x01
2022-10-03T02:15:49.4956353Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 1 byte 2 exp: 0x25 act: 0x02
2022-10-03T02:15:49.4956392Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 1 byte 3 exp: 0xcb act: 0x03
2022-10-03T02:15:49.4956432Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 2 byte 1 exp: 0xa8 act: 0x01
2022-10-03T02:15:49.4956471Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 2 byte 2 exp: 0x25 act: 0x02
2022-10-03T02:15:49.4956509Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 2 byte 3 exp: 0xcb act: 0x03
2022-10-03T02:15:49.4956550Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 3 byte 1 exp: 0xa8 act: 0x01
2022-10-03T02:15:49.4956588Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 3 byte 2 exp: 0x25 act: 0x02
2022-10-03T02:15:49.4956627Z - ERROR - FusionEventDriver - fct3: bad pad found: pad 7 bank 3 byte 3 exp: 0xcb act: 0x03
2022-10-03T02:15:49.7759994Z - INFO - FusionEventDriver - fct2: PCIe power monitor: disabled
2022-10-03T02:15:49.7760023Z - INFO - FusionEventDriver - fct2: Thermal monitoring: Enabled
2022-10-03T02:15:49.7760050Z - INFO - FusionEventDriver - fct2: Hardware temperature alarm set for 100C.
2022-10-03T02:15:49.9641956Z - ERROR - FusionEventDriver - fct3: MINIMAL MODE DRIVER: General channel initialization failure. error code:-1
2022-10-03T02:15:51.7182618Z - INFO - FusionEventDriver - fct2-1215D0142: Attempting to load ioDrive.
2022-10-03T02:15:51.8058296Z - INFO - FusionEventDriver - fct2-1215D0142: sector_size=512
2022-10-03T02:15:51.8058338Z - INFO - FusionEventDriver - fct2-1215D0142: setting channel range data to [2 .. 2047]
2022-10-03T02:15:52.2329291Z - INFO - FusionEventDriver - fct2-1215D0142: Found metadata in EBs 1487-1487, loading...
2022-10-03T02:15:52.7646225Z - INFO - FusionEventDriver - fct2-1215D0142: setting recovered append point 1487+749518848
2022-10-03T02:15:52.7860958Z - INFO - FusionEventDriver - fct2-1215D0142: Creating device of size 1205000000000 bytes with 2353515625 sectors of 512 bytes (1412115131 mapped).
2022-10-03T02:15:52.7860981Z - INFO - FusionEventDriver - fct2-1215D0142: Creating block device internal sector size: 512...
2022-10-03T02:15:52.7861221Z - INFO - FusionEventDriver - fct2-1215D0142: Attach succeeded.
2022-10-03T02:15:52.7861242Z - INFO - FusionEventDriver - fct2-1215D0142: ioDrive was successfully loaded and attached to the system.
2022-10-03T02:15:52.7866246Z - INFO - FusionEventDriver - Creating block device external sector size: 512...
2022-10-03T02:15:59.9656362Z - INFO - FusionEventDriver - fct3-1208D5248: IOCTL 0xc030684d unsupported in minimal mode
2022-10-03T02:15:59.9656425Z - INFO - FusionEventDriver - fiochkdrv: starting
2022-10-03T02:15:59.9656482Z - INFO - FusionEventDriver - fiochkdrv: no unattached drives found
2022-10-03T02:15:59.9656533Z - INFO - FusionEventDriver - fiochkdrv: stopping