LSI 9400-16i Power-on or device reset occurred

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
TrevorH

TrevorH

Active Member
Oct 25, 2024
222
96
28
I don't use TrueNAS but since it's linux based, check with e.g journalctl | grep -i timer to see if they use a systemd timer to trigger events. Also check /var/log/cron (location may vary).
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
Got new cables today, tried them out and no change. Still spams all sorts of the same errors during boot process.

Then I moved all the disks (I only have 8 at the moment) to the rear backplane and imagine my excitement when there are NO errors during the boot process. This is a fresh boot only minutes ago so maybe it will still have problems, but this is the only time I've ever seen it boot without any errors. Promising results anyways. It still has a few of the "power on device reset" errors but my understanding is this is normal to have during boot since things are initializing. Where this log ends my other ones continue with all the timeout errors and more resets.

Time to track down the seller of this chassis and see if he has another backplane I can buy. He was selling 100+ of these chassis at the time so I'm hopeful he still has some left.

Thanks again everyone for the help so far.

Code: 
Mar 12 20:38:38 vault kernel: scsi host0: Virtio SCSI HBA
Mar 12 20:38:38 vault kernel: scsi 0:0:0:0: Direct-Access     QEMU     QEMU HARDDISK    2.5+ PQ: 0 ANSI: 5
Mar 12 20:38:38 vault kernel: scsi host1: Virtio SCSI HBA
Mar 12 20:38:38 vault kernel: scsi 1:0:0:1: Direct-Access     QEMU     QEMU HARDDISK    2.5+ PQ: 0 ANSI: 5
Mar 12 20:38:38 vault kernel: sd 0:0:0:0: Power-on or device reset occurred
Mar 12 20:38:38 vault kernel: sd 1:0:0:1: Power-on or device reset occurred
Mar 12 20:38:38 vault kernel: sd 0:0:0:0: [sda] 67108864 512-byte logical blocks: (34.4 GB/32.0 GiB)
Mar 12 20:38:38 vault kernel: sd 1:0:0:1: [sdb] 67108864 512-byte logical blocks: (34.4 GB/32.0 GiB)
Mar 12 20:38:38 vault kernel: sd 0:0:0:0: [sda] Write Protect is off
Mar 12 20:38:38 vault kernel: sd 1:0:0:1: [sdb] Write Protect is off
Mar 12 20:38:38 vault kernel: sd 1:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:38 vault kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:38 vault kernel: mpt3sas version 48.100.00.00 loaded
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (16383524 kB)
Mar 12 20:38:38 vault kernel: sd 0:0:0:0: [sda] Attached SCSI disk
Mar 12 20:38:38 vault kernel: sd 1:0:0:1: [sdb] Attached SCSI disk
Mar 12 20:38:38 vault kernel: scsi host3: ahci
Mar 12 20:38:38 vault kernel: scsi host4: ahci
Mar 12 20:38:38 vault kernel: scsi host5: ahci
Mar 12 20:38:38 vault kernel: scsi host6: ahci
Mar 12 20:38:38 vault kernel: scsi host7: ahci
Mar 12 20:38:38 vault kernel: scsi host8: ahci
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: MSI-X vectors supported: 16
Mar 12 20:38:38 vault kernel: mpt3sas_cm0:  0 4 4
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: High IOPs queues : disabled
Mar 12 20:38:38 vault kernel: mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 46
Mar 12 20:38:38 vault kernel: mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 47
Mar 12 20:38:38 vault kernel: mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 48
Mar 12 20:38:38 vault kernel: mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 49
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: iomem(0x000000c0e0000000), mapped(0x000000006777ce75), size(1048576)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: ioport(0x0000000000005000), size(256)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: request pool(0x00000000f855c021) - dma(0x100400000): depth(6756), frame_size(128), pool_size(844 kB)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: sense pool(0x00000000b2e53fb6) - dma(0x100500000): depth(6551), element_size(96), pool_size (614 kB)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: reply pool(0x0000000003f02293) - dma(0x100600000): depth(6820), frame_size(128), pool_size(852 kB)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: config page(0x0000000070bc1fe9) - dma(0x153341000): size(512)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: Allocated physical memory: size(28966 kB)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: Current Controller Queue Depth(6548),Max Controller Queue Depth(6656)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: Scatter Gather Elements per IO(128)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: _base_display_fwpkg_version: complete
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: FW Package Ver(24.00.00.00)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: TimeSync Interval in Manuf page-11 is not enabled. Periodic Time-Sync will be disabled
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: SAS3416: FWVersion(24.00.00.00), ChipRevision(0x01)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Diag Trace Buffer,Task Set Full,NCQ)
Mar 12 20:38:38 vault kernel: mpt3sas 0000:01:00.0: Max SCSIIO MPT commands: 6548 shared with nr_hw_queues = 4
Mar 12 20:38:38 vault kernel: scsi host2: Fusion MPT SAS Host
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: sending port enable !!
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: hba_port entry: 000000005039f15f, port: 0 is added to hba_port list
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: hba_port entry: 000000002cc708ae, port: 8 is added to hba_port list
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00f3212d0), phys(21)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: expander_add: handle(0x0018), parent(0x0009), sas_addr(0x3474f4f584a11c3f), phys(37)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x1f) sas_address(0x3474f4f584a11c3e) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: expander_add: handle(0x0017), parent(0x0001), sas_addr(0x3474f4f58496a31f), phys(25)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x19) sas_address(0x3474f4f58496a300) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x1a) sas_address(0x3474f4f58496a301) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x1b) sas_address(0x3474f4f58496a302) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x1c) sas_address(0x3474f4f58496a303) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x1d) sas_address(0x3474f4f58496a307) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x1e) sas_address(0x3474f4f58496a308) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x20) sas_address(0x3474f4f58496a31e) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x21) sas_address(0x3474f4f58496a304) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: handle(0x22) sas_address(0x3474f4f58496a305) port_type(0x1)
Mar 12 20:38:38 vault kernel: mpt3sas_cm0: port enable: SUCCESS
Mar 12 20:38:38 vault kernel: scsi 2:0:0:0: Enclosure         GOOXIBM  4U24SXP 36Sx12G  B134 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:0:0: set ignore_delay_remove for handle(0x001f)
Mar 12 20:38:38 vault kernel: scsi 2:0:0:0: SES: handle(0x001f), sas_addr(0x3474f4f584a11c3e), phy(36), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:0:0: enclosure logical id (0x3474f4f584a11c3e), slot(24)
Mar 12 20:38:38 vault kernel: scsi 2:0:0:0: enclosure level(0x0000), connector name( C0  )
Mar 12 20:38:38 vault kernel: scsi 2:0:0:0: qdepth(64), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:1:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:1:0: SATA: handle(0x0019), sas_addr(0x3474f4f58496a300), phy(0), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:1:0: enclosure logical id (0x3474f4f58496a300), slot(0)
Mar 12 20:38:38 vault kernel: scsi 2:0:1:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:38 vault kernel: scsi 2:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 12 20:38:38 vault kernel: scsi 2:0:1:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:2:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:2:0: SATA: handle(0x001a), sas_addr(0x3474f4f58496a301), phy(1), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:2:0: enclosure logical id (0x3474f4f58496a300), slot(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:2:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:38 vault kernel: scsi 2:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 12 20:38:38 vault kernel: scsi 2:0:2:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:3:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:3:0: SATA: handle(0x001b), sas_addr(0x3474f4f58496a302), phy(2), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:3:0: enclosure logical id (0x3474f4f58496a300), slot(2)
Mar 12 20:38:38 vault kernel: scsi 2:0:3:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:38 vault kernel: scsi 2:0:3:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 12 20:38:38 vault kernel: scsi 2:0:3:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:4:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:4:0: SATA: handle(0x001c), sas_addr(0x3474f4f58496a303), phy(3), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:4:0: enclosure logical id (0x3474f4f58496a300), slot(3)
Mar 12 20:38:38 vault kernel: scsi 2:0:4:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:38 vault kernel: scsi 2:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 12 20:38:38 vault kernel: scsi 2:0:4:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:5:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:5:0: SATA: handle(0x001d), sas_addr(0x3474f4f58496a307), phy(7), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:5:0: enclosure logical id (0x3474f4f58496a300), slot(7)
Mar 12 20:38:38 vault kernel: scsi 2:0:5:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:38 vault kernel: scsi 2:0:5:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 12 20:38:38 vault kernel: scsi 2:0:5:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:6:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:6:0: SATA: handle(0x001e), sas_addr(0x3474f4f58496a308), phy(8), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:6:0: enclosure logical id (0x3474f4f58496a300), slot(8)
Mar 12 20:38:38 vault kernel: scsi 2:0:6:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:38 vault kernel: scsi 2:0:6:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 12 20:38:38 vault kernel: scsi 2:0:6:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:7:0: Enclosure         GOOXIBM  2U12SXP 24Sx12G  B134 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:7:0: set ignore_delay_remove for handle(0x0020)
Mar 12 20:38:38 vault kernel: scsi 2:0:7:0: SES: handle(0x0020), sas_addr(0x3474f4f58496a31e), phy(24), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:7:0: enclosure logical id (0x3474f4f58496a300), slot(39)
Mar 12 20:38:38 vault kernel: scsi 2:0:7:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:38 vault kernel: scsi 2:0:7:0: qdepth(64), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:38 vault kernel: scsi 2:0:7:0: Power-on or device reset occurred
Mar 12 20:38:38 vault kernel: scsi 2:0:8:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 12 20:38:38 vault kernel: scsi 2:0:8:0: SATA: handle(0x0021), sas_addr(0x3474f4f58496a304), phy(4), device_name(0x0000000000000000)
Mar 12 20:38:38 vault kernel: scsi 2:0:8:0: enclosure logical id (0x3474f4f58496a300), slot(4)
Mar 12 20:38:38 vault kernel: scsi 2:0:8:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:38 vault kernel: scsi 2:0:8:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 12 20:38:39 vault kernel: scsi 2:0:8:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:39 vault kernel: scsi 2:0:9:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 12 20:38:39 vault kernel: scsi 2:0:9:0: SATA: handle(0x0022), sas_addr(0x3474f4f58496a305), phy(5), device_name(0x0000000000000000)
Mar 12 20:38:39 vault kernel: scsi 2:0:9:0: enclosure logical id (0x3474f4f58496a300), slot(5)
Mar 12 20:38:39 vault kernel: scsi 2:0:9:0: enclosure level(0x0000), connector name( C3  )
Mar 12 20:38:39 vault kernel: scsi 2:0:9:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 12 20:38:39 vault kernel: scsi 2:0:9:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 12 20:38:39 vault kernel: sd 2:0:1:0: Power-on or device reset occurred
Mar 12 20:38:39 vault kernel: sd 2:0:2:0: Power-on or device reset occurred
Mar 12 20:38:39 vault kernel: sd 2:0:1:0: [sdc] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 12 20:38:39 vault kernel: sd 2:0:3:0: Power-on or device reset occurred
Mar 12 20:38:39 vault kernel: scsi 4:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
Mar 12 20:38:39 vault kernel: sd 2:0:2:0: [sdd] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 12 20:38:39 vault kernel: sd 2:0:2:0: [sdd] 4096-byte physical blocks
Mar 12 20:38:39 vault kernel: sd 2:0:1:0: [sdc] 4096-byte physical blocks
Mar 12 20:38:39 vault kernel: sd 2:0:3:0: [sde] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 12 20:38:39 vault kernel: sd 2:0:4:0: Power-on or device reset occurred
Mar 12 20:38:39 vault kernel: sd 2:0:3:0: [sde] 4096-byte physical blocks
Mar 12 20:38:39 vault kernel: sd 2:0:1:0: [sdc] Write Protect is off
Mar 12 20:38:39 vault kernel: sd 2:0:5:0: Power-on or device reset occurred
Mar 12 20:38:39 vault kernel: sd 2:0:6:0: Power-on or device reset occurred
Mar 12 20:38:39 vault kernel: sd 2:0:8:0: Power-on or device reset occurred
Mar 12 20:38:39 vault kernel: sd 2:0:9:0: Power-on or device reset occurred
Mar 12 20:38:39 vault kernel: sd 2:0:1:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:39 vault kernel: sd 2:0:4:0: [sdf] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 12 20:38:39 vault kernel: sd 2:0:4:0: [sdf] 4096-byte physical blocks
Mar 12 20:38:39 vault kernel: sd 2:0:2:0: [sdd] Write Protect is off
Mar 12 20:38:39 vault kernel: sd 2:0:8:0: [sdi] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 12 20:38:39 vault kernel: sd 2:0:8:0: [sdi] 4096-byte physical blocks
Mar 12 20:38:39 vault kernel: sd 2:0:9:0: [sdj] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 12 20:38:39 vault kernel: sd 2:0:6:0: [sdh] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 12 20:38:39 vault kernel: sd 2:0:6:0: [sdh] 4096-byte physical blocks
Mar 12 20:38:39 vault kernel: sd 2:0:3:0: [sde] Write Protect is off
Mar 12 20:38:39 vault kernel: sd 2:0:5:0: [sdg] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 12 20:38:39 vault kernel: sd 2:0:5:0: [sdg] 4096-byte physical blocks
Mar 12 20:38:39 vault kernel: sd 2:0:9:0: [sdj] 4096-byte physical blocks
Mar 12 20:38:39 vault kernel: sd 2:0:3:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:39 vault kernel: sd 2:0:2:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:39 vault kernel: sd 2:0:4:0: [sdf] Write Protect is off
Mar 12 20:38:39 vault kernel: sd 2:0:9:0: [sdj] Write Protect is off
Mar 12 20:38:39 vault kernel: sd 2:0:6:0: [sdh] Write Protect is off
Mar 12 20:38:39 vault kernel: sd 2:0:8:0: [sdi] Write Protect is off
Mar 12 20:38:39 vault kernel: sd 2:0:5:0: [sdg] Write Protect is off
Mar 12 20:38:39 vault kernel: sd 2:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:39 vault kernel: sd 2:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:39 vault kernel: sd 2:0:6:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:39 vault kernel: sd 2:0:4:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:39 vault kernel: sd 2:0:5:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 12 20:38:39 vault kernel: sd 2:0:1:0: [sdc] Attached SCSI disk
Mar 12 20:38:39 vault kernel: sd 2:0:4:0: [sdf] Attached SCSI disk
Mar 12 20:38:39 vault kernel: sd 2:0:6:0: [sdh] Attached SCSI disk
Mar 12 20:38:39 vault kernel: sd 2:0:2:0: [sdd] Attached SCSI disk
Mar 12 20:38:39 vault kernel: sd 2:0:3:0: [sde] Attached SCSI disk
Mar 12 20:38:39 vault kernel: sd 2:0:8:0: [sdi] Attached SCSI disk
Mar 12 20:38:39 vault kernel: sd 2:0:5:0: [sdg] Attached SCSI disk
Mar 12 20:38:39 vault kernel: sd 2:0:9:0: [sdj] Attached SCSI disk
Mar 12 20:38:39 vault kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
Mar 12 20:38:39 vault kernel: sd 1:0:0:1: Attached scsi generic sg1 type 0
Mar 12 20:38:39 vault kernel: ses 2:0:0:0: Attached scsi generic sg2 type 13
Mar 12 20:38:39 vault kernel: sd 2:0:1:0: Attached scsi generic sg3 type 0
Mar 12 20:38:39 vault kernel: sd 2:0:2:0: Attached scsi generic sg4 type 0
Mar 12 20:38:39 vault kernel: sd 2:0:3:0: Attached scsi generic sg5 type 0
Mar 12 20:38:39 vault kernel: sd 2:0:4:0: Attached scsi generic sg6 type 0
Mar 12 20:38:39 vault kernel: sd 2:0:5:0: Attached scsi generic sg7 type 0
Mar 12 20:38:39 vault kernel: sd 2:0:6:0: Attached scsi generic sg8 type 0
Mar 12 20:38:39 vault kernel: ses 2:0:7:0: Attached scsi generic sg9 type 13
Mar 12 20:38:39 vault kernel: sd 2:0:8:0: Attached scsi generic sg10 type 0
Mar 12 20:38:39 vault kernel: sd 2:0:9:0: Attached scsi generic sg11 type 0
Mar 12 20:38:39 vault kernel: sr 4:0:0:0: Attached scsi generic sg12 type 5
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
TrevorH said:
I don't use TrueNAS but since it's linux based, check with e.g journalctl | grep -i timer to see if they use a systemd timer to trigger events. Also check /var/log/cron (location may vary).
Click to expand...
Thanks Trevor, I did look there already and although there are many timers being set and such, I don't see any on a 90 min interval. I guess it could be a 30 min interval where it does something differently every 3rd run ... or something weird like that. But as per my last message above, I am optimistic the problem is the actual backplane. Maybe it has an internal 90min timer that is doing something? No idea really. But if the errors go away then I don't need to understand why other than it being a bad backplane.

Got in contact with the seller, he has more sitting around and has offered to let me take one to swap the backplanes and test it out. Aiming to do that this weekend and will report back.
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
Well the plot thickens. I picked up a second of this chassis today and swapped the backplanes. Made no difference at all - the problem is exactly the same.

Did some more testing with cable arrangements too - no matter what I do the drives only work without errors in the back 12-port backplane. Even if I configure it in series (2x cable from HBA to 24-port backplane, 1x cable from 24-port to 12-port backplane), the drives work perfectly fine when connected to the 12-port.

I've reached out to Gooxi to see if there are any firmware updates for this backplane. Not sure what else to try at this point. Could try another chassis I suppose but he says he's sold 40x of these used chassis and I'm the first person to report any issues with them so it seems unlikely that I got 2x bad ones ...

Maybe for testing's sake I need to install TrueNAS on bare metal instead of virtualized just to see if that makes a difference? That might not be all that difficult to do I suppose.
 
E

EasyRhino

Well-Known Member
Aug 6, 2019
696
579
93
To chime, in, I don't think it applies to your particular details, but I had a lot of similar sounding device reset messages from truenas with my 9400-16i... but it wasn't the card at all, I had a failing SATA power splitter cable on the drives.
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
EDIT: after posting this I realized that my sdc drive has actually faulted in TrueNAS UI (all the errors are related to it as well). So maybe this latest set of errors is just due to a failing drive? Will investigate tonight. I have a spare so I can swap that out.


Alright I have some updates to this saga. I'm just going to write this all out so you can see what I tried, sorry if it gets a bit long :)

1. I found an old SSD, swapped it into the system, installed TrueNAS, imported config from the VM. I was pleased to see no errors upon booting. At this point in time I believed that the problem was 100% reproducible by simply booting the system (turns out I was wrong, more on that later). After some quick testing on bare metal I hypothesized that the problem was solved using bare metal and that the problem must be in the passthrough of the HBA.

2. So I booted back into Proxmox and started playing with passthrough settings. The main one being adding "iommu=pt" to proxmox boot cmdline because I hadn't done this before. Note that passthrough was seemingly always working but not knowing enough about how this all works I figured it was worth a try. Imagine my surprise and excitement when upon booting there were no errors to be seen anywhere! (same as bare metal). I even left this for over 24hrs and still no errors anywhere. I was a bit confused by this as to what it was actually doing because I couldn't see any difference in the boot process or detection of devices with and without the option, but if it works it works!

3. To prove to myself that this actually fixed the problem, I reverted the setting back to default and rebooted the system. Errors came back! Looking good!

4. Added the option back in, booted, and no errors again! woohoo!

5. Let's try this again, removed the option, booted, errors all over the place!

6. Great, problem is solved, right? WRONG! I put the option back in one final time, booted and ... errors everywhere. Reboot a bunch of times with option in exactly as in (2) and (4) above and see that most of the boots it has errors, but occasionally it doesn't. Okay, so booting is not 100% way to reproduce the issue.

7. Left it alone overnight after booting once without errors. Found errors in the morning that happened 90 minutes after boot. So even if it boots without errors, they'll still happen eventually (as before, on a multiple of 90 minutes from last boot time).

8. Okay, so back to square one. Let's try bare metal again. Boot to that SSD again, and this time it's errors galore. Way more than I ever saw in the VM and different ones.

Open to any thoughts. Things I will try soon:
  1. Different power supply - I have a second I can pull from the other chassis. This will eliminate it being a power issue.
  2. Different HBA - I have a 9300-8i on the way
  3. HBA overheating - I don't think it's this, there is good airflow over the card and it doesn't feel overly hot to the touch. But maybe I'll throw a fan on it just to be 100% sure.
As a refresher, here's the things I have already tried that didn't make a difference:
  1. Brand new cable
  2. Different cable arrangements (different HBA ports and different backplane ports)
  3. Backplane from another identical chassis
  4. Rear backplane works fine using same cable and HBA ports (maybe I should confirm that on bare metal though)
  5. 9400-16i HBA is up to date for firmware
  6. Different PCIe slot for the HBA

Code: 
Mar 17 07:37:17 vault kernel: mpt3sas version 48.100.00.00 loaded
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (65703400 kB)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: MSI-X vectors supported: 128
Mar 17 07:37:17 vault kernel: mpt3sas_cm0:  0 32 32
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: High IOPs queues : disabled
Mar 17 07:37:17 vault kernel: mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 82
Mar 17 07:37:17 vault kernel: mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 83
Mar 17 07:37:17 vault kernel: mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 84
Mar 17 07:37:17 vault kernel: mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 85
Mar 17 07:37:17 vault kernel: mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 86
Mar 17 07:37:17 vault kernel: mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 87
Mar 17 07:37:17 vault kernel: mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 88
Mar 17 07:37:17 vault kernel: mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 89
Mar 17 07:37:17 vault kernel: mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 90
Mar 17 07:37:17 vault kernel: mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 91
Mar 17 07:37:17 vault kernel: mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 92
Mar 17 07:37:17 vault kernel: mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 93
Mar 17 07:37:17 vault kernel: mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 94
Mar 17 07:37:17 vault kernel: mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 95
Mar 17 07:37:17 vault kernel: mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 96
Mar 17 07:37:17 vault kernel: mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 97
Mar 17 07:37:17 vault kernel: mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 98
Mar 17 07:37:17 vault kernel: mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 99
Mar 17 07:37:17 vault kernel: mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 100
Mar 17 07:37:17 vault kernel: mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 101
Mar 17 07:37:17 vault kernel: mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 102
Mar 17 07:37:17 vault kernel: mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 103
Mar 17 07:37:17 vault kernel: mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 104
Mar 17 07:37:17 vault kernel: mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 105
Mar 17 07:37:17 vault kernel: mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 106
Mar 17 07:37:17 vault kernel: mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 107
Mar 17 07:37:17 vault kernel: mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 108
Mar 17 07:37:17 vault kernel: mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 109
Mar 17 07:37:17 vault kernel: mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 110
Mar 17 07:37:17 vault kernel: mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 111
Mar 17 07:37:17 vault kernel: mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 112
Mar 17 07:37:17 vault kernel: mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 113
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: iomem(0x0000028080f00000), mapped(0x00000000bb147231), size(1048576)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: ioport(0x0000000000007000), size(256)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: sending message unit reset !!
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: message unit reset: SUCCESS
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: request pool(0x000000003c65a521) - dma(0x100c00000): depth(6756), frame_size(128), pool_size(844 kB)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: sense pool(0x00000000be8a14aa) - dma(0x100d00000): depth(6551), element_size(96), pool_size (614 kB)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: reply pool(0x0000000010d0eb6c) - dma(0x100e00000): depth(6820), frame_size(128), pool_size(852 kB)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: config page(0x000000008ec4994c) - dma(0x12dbbe000): size(512)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: Allocated physical memory: size(31938 kB)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: Current Controller Queue Depth(6548),Max Controller Queue Depth(6656)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: Scatter Gather Elements per IO(128)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: _base_display_fwpkg_version: complete
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: FW Package Ver(24.00.00.00)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: TimeSync Interval in Manuf page-11 is not enabled. Periodic Time-Sync will be disabled
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: SAS3416: FWVersion(24.00.00.00), ChipRevision(0x01)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Diag Trace Buffer,Task Set Full,NCQ)
Mar 17 07:37:17 vault kernel: mpt3sas 0000:41:00.0: Max SCSIIO MPT commands: 6548 shared with nr_hw_queues = 32
Mar 17 07:37:17 vault kernel: scsi host0: Fusion MPT SAS Host
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: sending port enable !!
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: hba_port entry: 0000000026c25a9c, port: 0 is added to hba_port list
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: hba_port entry: 000000008d4d5554, port: 8 is added to hba_port list
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00f3212d0), phys(21)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: expander_add: handle(0x0017), parent(0x0009), sas_addr(0x3474f4f584b3483f), phys(37)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x18) sas_address(0x3474f4f584b34808) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x19) sas_address(0x3474f4f584b34809) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x1a) sas_address(0x3474f4f584b3480a) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x1b) sas_address(0x3474f4f584b3480b) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x1c) sas_address(0x3474f4f584b3480c) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x1d) sas_address(0x3474f4f584b3480d) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x1e) sas_address(0x3474f4f584b3480e) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x1f) sas_address(0x3474f4f584b3480f) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: handle(0x20) sas_address(0x3474f4f584b3483e) port_type(0x1)
Mar 17 07:37:17 vault kernel: mpt3sas_cm0: port enable: SUCCESS
Mar 17 07:37:17 vault kernel: scsi 0:0:0:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:0:0: SATA: handle(0x0018), sas_addr(0x3474f4f584b34808), phy(8), device_name(0x5000c500e58ea085)
Mar 17 07:37:17 vault kernel: scsi 0:0:0:0: enclosure logical id (0x3474f4f584b3483e), slot(8)
Mar 17 07:37:17 vault kernel: scsi 0:0:0:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 17 07:37:17 vault kernel: scsi 0:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: scsi 0:0:1:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:1:0: SATA: handle(0x0019), sas_addr(0x3474f4f584b34809), phy(9), device_name(0x5000c500e59a40b4)
Mar 17 07:37:17 vault kernel: scsi 0:0:1:0: enclosure logical id (0x3474f4f584b3483e), slot(9)
Mar 17 07:37:17 vault kernel: scsi 0:0:1:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 17 07:37:17 vault kernel: scsi 0:0:1:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: scsi 0:0:2:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:2:0: SATA: handle(0x001a), sas_addr(0x3474f4f584b3480a), phy(10), device_name(0x5000c500e54223b7)
Mar 17 07:37:17 vault kernel: scsi 0:0:2:0: enclosure logical id (0x3474f4f584b3483e), slot(10)
Mar 17 07:37:17 vault kernel: scsi 0:0:2:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 17 07:37:17 vault kernel: scsi 0:0:2:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: scsi 0:0:3:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:3:0: SATA: handle(0x001b), sas_addr(0x3474f4f584b3480b), phy(11), device_name(0x5000c500e59afd90)
Mar 17 07:37:17 vault kernel: scsi 0:0:3:0: enclosure logical id (0x3474f4f584b3483e), slot(11)
Mar 17 07:37:17 vault kernel: scsi 0:0:3:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:3:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 17 07:37:17 vault kernel: scsi 0:0:3:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: scsi 0:0:4:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:4:0: SATA: handle(0x001c), sas_addr(0x3474f4f584b3480c), phy(12), device_name(0x5000c500e5aae301)
Mar 17 07:37:17 vault kernel: scsi 0:0:4:0: enclosure logical id (0x3474f4f584b3483e), slot(12)
Mar 17 07:37:17 vault kernel: scsi 0:0:4:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 17 07:37:17 vault kernel: scsi 0:0:4:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: scsi 0:0:5:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:5:0: SATA: handle(0x001d), sas_addr(0x3474f4f584b3480d), phy(13), device_name(0x5000c500e59fee3f)
Mar 17 07:37:17 vault kernel: scsi 0:0:5:0: enclosure logical id (0x3474f4f584b3483e), slot(13)
Mar 17 07:37:17 vault kernel: scsi 0:0:5:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:5:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 17 07:37:17 vault kernel: scsi 0:0:5:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: scsi 0:0:6:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:6:0: SATA: handle(0x001e), sas_addr(0x3474f4f584b3480e), phy(14), device_name(0x5000c500e5904dd0)
Mar 17 07:37:17 vault kernel: scsi 0:0:6:0: enclosure logical id (0x3474f4f584b3483e), slot(14)
Mar 17 07:37:17 vault kernel: scsi 0:0:6:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:6:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 17 07:37:17 vault kernel: scsi 0:0:6:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: scsi 0:0:7:0: Direct-Access     ATA      ST20000NM007D-3D SN06 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:7:0: SATA: handle(0x001f), sas_addr(0x3474f4f584b3480f), phy(15), device_name(0x5000c500e5aa4fc8)
Mar 17 07:37:17 vault kernel: scsi 0:0:7:0: enclosure logical id (0x3474f4f584b3483e), slot(15)
Mar 17 07:37:17 vault kernel: scsi 0:0:7:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Mar 17 07:37:17 vault kernel: scsi 0:0:7:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: scsi 0:0:8:0: Enclosure         GOOXIBM  4U24SXP 36Sx12G  B134 PQ: 0 ANSI: 6
Mar 17 07:37:17 vault kernel: scsi 0:0:8:0: set ignore_delay_remove for handle(0x0020)
Mar 17 07:37:17 vault kernel: scsi 0:0:8:0: SES: handle(0x0020), sas_addr(0x3474f4f584b3483e), phy(36), device_name(0x0000000000000000)
Mar 17 07:37:17 vault kernel: scsi 0:0:8:0: enclosure logical id (0x3474f4f584b3483e), slot(24)
Mar 17 07:37:17 vault kernel: scsi 0:0:8:0: enclosure level(0x0000), connector name( C0  )
Mar 17 07:37:17 vault kernel: scsi 0:0:8:0: qdepth(64), tagged(1), scsi_level(7), cmd_que(1)
Mar 17 07:37:17 vault kernel: sd 0:0:1:0: [sda] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 17 07:37:17 vault kernel: sd 0:0:0:0: [sdb] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 17 07:37:17 vault kernel: sd 0:0:2:0: [sdc] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 17 07:37:17 vault kernel: sd 0:0:0:0: [sdb] 4096-byte physical blocks
Mar 17 07:37:17 vault kernel: sd 0:0:4:0: [sde] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 17 07:37:17 vault kernel: sd 0:0:4:0: [sde] 4096-byte physical blocks
Mar 17 07:37:17 vault kernel: sd 0:0:3:0: [sdd] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 17 07:37:17 vault kernel: sd 0:0:6:0: [sdg] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 17 07:37:17 vault kernel: sd 0:0:3:0: [sdd] 4096-byte physical blocks
Mar 17 07:37:17 vault kernel: sd 0:0:6:0: [sdg] 4096-byte physical blocks
Mar 17 07:37:17 vault kernel: sd 0:0:0:0: [sdb] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 0:0:6:0: [sdg] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 0:0:4:0: [sde] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 0:0:3:0: [sdd] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 0:0:5:0: [sdf] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 17 07:37:17 vault kernel: sd 0:0:7:0: [sdh] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Mar 17 07:37:17 vault kernel: sd 0:0:7:0: [sdh] 4096-byte physical blocks
Mar 17 07:37:17 vault kernel: sd 0:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 17 07:37:17 vault kernel: sd 0:0:1:0: [sda] 4096-byte physical blocks
Mar 17 07:37:17 vault kernel: sd 0:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 17 07:37:17 vault kernel: sd 0:0:7:0: [sdh] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 0:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 17 07:37:17 vault kernel: sd 0:0:1:0: [sda] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 0:0:2:0: [sdc] 4096-byte physical blocks
Mar 17 07:37:17 vault kernel: sd 0:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 17 07:37:17 vault kernel: sd 0:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 17 07:37:17 vault kernel: sd 0:0:5:0: [sdf] 4096-byte physical blocks
Mar 17 07:37:17 vault kernel: sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 17 07:37:17 vault kernel: sd 0:0:5:0: [sdf] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 0:0:2:0: [sdc] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 0:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 17 07:37:17 vault kernel: sd 0:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
Mar 17 07:37:17 vault kernel: sd 0:0:7:0: [sdh] Attached SCSI disk
Mar 17 07:37:17 vault kernel: sd 0:0:0:0: [sdb] Attached SCSI disk
Mar 17 07:37:17 vault kernel: sd 0:0:6:0: [sdg] Attached SCSI disk
Mar 17 07:37:17 vault kernel: sd 0:0:1:0: [sda] Attached SCSI disk
Mar 17 07:37:17 vault kernel: sd 0:0:5:0: [sdf] Attached SCSI disk
Mar 17 07:37:17 vault kernel: sd 0:0:4:0: [sde] Attached SCSI disk
Mar 17 07:37:17 vault kernel: sd 0:0:3:0: [sdd] Attached SCSI disk
Mar 17 07:37:17 vault kernel: sd 0:0:2:0: [sdc] Attached SCSI disk
Mar 17 07:37:17 vault kernel: scsi host1: ahci
Mar 17 07:37:17 vault kernel: scsi host2: ahci
Mar 17 07:37:17 vault kernel: scsi host3: ahci
Mar 17 07:37:17 vault kernel: scsi host4: ahci
Mar 17 07:37:17 vault kernel: scsi host5: ahci
Mar 17 07:37:17 vault kernel: scsi host6: ahci
Mar 17 07:37:17 vault kernel: scsi host7: ahci
Mar 17 07:37:17 vault kernel: scsi host8: ahci
Mar 17 07:37:17 vault kernel: scsi host9: ahci
Mar 17 07:37:17 vault kernel: scsi host10: ahci
Mar 17 07:37:17 vault kernel: scsi host11: ahciem
Mar 17 07:37:17 vault kernel: scsi 11:0:0:0: Enclosure         AHCI     SGPIO Enclosure  2.00 PQ: 0 ANSI: 7
Mar 17 07:37:17 vault kernel: scsi host12: ahci
Mar 17 07:37:17 vault kernel: scsi host13: ahci
Mar 17 07:37:17 vault kernel: scsi host14: ahci
Mar 17 07:37:17 vault kernel: scsi host15: ahci
Mar 17 07:37:17 vault kernel: scsi host16: ahci
Mar 17 07:37:17 vault kernel: scsi host17: ahci
Mar 17 07:37:17 vault kernel: scsi host18: ahci
Mar 17 07:37:17 vault kernel: scsi host19: ahci
Mar 17 07:37:17 vault kernel: scsi host20: ahciem
Mar 17 07:37:17 vault kernel: scsi 20:0:0:0: Enclosure         AHCI     SGPIO Enclosure  2.00 PQ: 0 ANSI: 7
Mar 17 07:37:17 vault kernel: scsi 14:0:0:0: Direct-Access     ATA      Samsung SSD 870  1B6Q PQ: 0 ANSI: 5
Mar 17 07:37:17 vault kernel: sd 14:0:0:0: [sdi] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
Mar 17 07:37:17 vault kernel: sd 14:0:0:0: [sdi] Write Protect is off
Mar 17 07:37:17 vault kernel: sd 14:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Mar 17 07:37:17 vault kernel: sd 14:0:0:0: [sdi] Preferred minimum I/O size 512 bytes
Mar 17 07:37:17 vault kernel: sd 14:0:0:0: [sdi] supports TCG Opal
Mar 17 07:37:17 vault kernel: sd 14:0:0:0: [sdi] Attached SCSI disk
Mar 17 07:37:18 vault kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
Mar 17 07:37:18 vault kernel: sd 0:0:1:0: Attached scsi generic sg1 type 0
Mar 17 07:37:18 vault kernel: sd 0:0:2:0: Attached scsi generic sg2 type 0
Mar 17 07:37:18 vault kernel: sd 0:0:3:0: Attached scsi generic sg3 type 0
Mar 17 07:37:18 vault kernel: sd 0:0:4:0: Attached scsi generic sg4 type 0
Mar 17 07:37:18 vault kernel: sd 0:0:5:0: Attached scsi generic sg5 type 0
Mar 17 07:37:18 vault kernel: sd 0:0:6:0: Attached scsi generic sg6 type 0
Mar 17 07:37:18 vault kernel: sd 0:0:7:0: Attached scsi generic sg7 type 0
Mar 17 07:37:18 vault kernel: ses 0:0:8:0: Attached scsi generic sg8 type 13
Mar 17 07:37:18 vault kernel: ses 11:0:0:0: Attached scsi generic sg9 type 13
Mar 17 07:37:18 vault kernel: ses 20:0:0:0: Attached scsi generic sg10 type 13
Mar 17 07:37:18 vault kernel: sd 14:0:0:0: Attached scsi generic sg11 type 0
Mar 17 07:38:59 vault kernel: mpt3sas_cm0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c)
Mar 17 07:38:59 vault kernel: mpt3sas_cm0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c)
Mar 17 07:38:59 vault kernel: mpt3sas_cm0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c)
Mar 17 07:38:59 vault kernel: mpt3sas_cm0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c)
Mar 17 07:38:59 vault kernel: mpt3sas_cm0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c)
Mar 17 07:38:59 vault kernel: mpt3sas_cm0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c)
Mar 17 07:39:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:18 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:19 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: attempting task abort!scmd(0x000000005e2d36e9), outstanding for 30244 ms & timeout 30000 ms
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: [sdc] tag#2455 CDB: Read(16) 88 00 00 00 00 01 c9 81 6c 18 00 00 00 08 00 00
Mar 17 07:39:29 vault kernel: scsi target0:0:2: handle(0x001a), sas_address(0x3474f4f584b3480a), phy(10)
Mar 17 07:39:29 vault kernel: scsi target0:0:2: enclosure logical id(0x3474f4f584b3483e), slot(10)
Mar 17 07:39:29 vault kernel: scsi target0:0:2: enclosure level(0x0000), connector name( C0  )
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: task abort: SUCCESS scmd(0x000000005e2d36e9)
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: attempting task abort!scmd(0x00000000c9bcc24e), outstanding for 30340 ms & timeout 30000 ms
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: [sdc] tag#2454 CDB: Read(16) 88 00 00 00 00 05 38 00 39 28 00 00 00 08 00 00
Mar 17 07:39:29 vault kernel: scsi target0:0:2: handle(0x001a), sas_address(0x3474f4f584b3480a), phy(10)
Mar 17 07:39:29 vault kernel: scsi target0:0:2: enclosure logical id(0x3474f4f584b3483e), slot(10)
Mar 17 07:39:29 vault kernel: scsi target0:0:2: enclosure level(0x0000), connector name( C0  )
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: No reference found at driver, assuming scmd(0x00000000c9bcc24e) might have completed
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: task abort: SUCCESS scmd(0x00000000c9bcc24e)
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: attempting task abort!scmd(0x00000000960c4f22), outstanding for 30336 ms & timeout 30000 ms
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: [sdc] tag#2453 CDB: Read(16) 88 00 00 00 00 06 c2 c2 70 90 00 00 00 08 00 00
Mar 17 07:39:29 vault kernel: scsi target0:0:2: handle(0x001a), sas_address(0x3474f4f584b3480a), phy(10)
Mar 17 07:39:29 vault kernel: scsi target0:0:2: enclosure logical id(0x3474f4f584b3483e), slot(10)
Mar 17 07:39:29 vault kernel: scsi target0:0:2: enclosure level(0x0000), connector name( C0  )
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: No reference found at driver, assuming scmd(0x00000000960c4f22) might have completed
Mar 17 07:39:29 vault kernel: sd 0:0:2:0: task abort: SUCCESS scmd(0x00000000960c4f22)
Mar 17 07:39:30 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:30 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:32 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:34 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:34 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:34 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:34 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:34 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:34 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:35 vault kernel: sd 0:0:2:0: [sdc] tag#2454 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=35s
Mar 17 07:39:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:35 vault kernel: sd 0:0:2:0: [sdc] tag#2454 CDB: Read(16) 88 00 00 00 00 05 38 00 39 28 00 00 00 08 00 00
Mar 17 07:39:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: sd 0:0:2:0: [sdc] tag#2462 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=36s
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: sd 0:0:2:0: [sdc] tag#2462 CDB: Read(16) 88 00 00 00 00 06 c2 c2 70 90 00 00 00 08 00 00
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: sd 0:0:2:0: [sdc] tag#2461 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=36s
Mar 17 07:39:36 vault kernel: sd 0:0:2:0: [sdc] tag#2461 CDB: Read(16) 88 00 00 00 00 01 c9 81 6c 18 00 00 00 08 00 00
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:37 vault kernel: sd 0:0:2:0: [sdc] tag#2465 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
Mar 17 07:39:37 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:37 vault kernel: sd 0:0:2:0: [sdc] tag#2463 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
Mar 17 07:39:37 vault kernel: sd 0:0:2:0: [sdc] tag#2465 CDB: Write(16) 8a 00 00 00 00 06 b9 c0 29 a8 00 00 00 08 00 00
Mar 17 07:39:37 vault kernel: sd 0:0:2:0: [sdc] tag#2463 CDB: Write(16) 8a 00 00 00 00 05 bd 00 28 a8 00 00 00 08 00 00
Mar 17 07:39:37 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:37 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:38 vault kernel: sd 0:0:2:0: [sdc] tag#2466 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=38s
Mar 17 07:39:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:38 vault kernel: sd 0:0:2:0: [sdc] tag#2466 CDB: Read(16) 88 00 00 00 00 01 c9 85 4b 50 00 00 00 08 00 00
Mar 17 07:39:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#3659 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=6s
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#3659 CDB: Read(16) 88 00 00 00 00 00 00 00 0a 10 00 00 00 10 00 00
Mar 17 07:39:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#2471 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=5s
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#2471 CDB: Read(16) 88 00 00 00 00 01 c9 80 31 60 00 00 00 08 00 00
Mar 17 07:39:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#3658 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#3658 CDB: Read(16) 88 00 00 00 00 09 18 20 04 10 00 00 00 10 00 00
Mar 17 07:39:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#3657 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#3657 CDB: Read(16) 88 00 00 00 00 09 18 20 06 10 00 00 00 10 00 00
Mar 17 07:39:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#3660 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
Mar 17 07:39:42 vault kernel: sd 0:0:2:0: [sdc] tag#3660 CDB: Read(16) 88 00 00 00 00 06 c2 c7 5c b0 00 00 00 08 00 00
Mar 17 07:39:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:43 vault kernel: sd 0:0:2:0: [sdc] tag#3447 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=5s
Mar 17 07:39:43 vault kernel: sd 0:0:2:0: [sdc] tag#4160 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=6s
Mar 17 07:39:43 vault kernel: sd 0:0:2:0: [sdc] tag#3447 CDB: Write(16) 8a 00 00 00 00 06 59 c0 28 90 00 00 00 10 00 00
Mar 17 07:39:43 vault kernel: sd 0:0:2:0: [sdc] tag#4160 CDB: Write(16) 8a 00 00 00 00 05 bd 00 28 b0 00 00 00 18 00 00
Mar 17 07:39:43 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:43 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:44 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:45 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:46 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:47 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:48 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:49 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:50 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:51 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:52 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:53 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:54 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:55 vault kernel: sd 0:0:2:0: [sdc] tag#962 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=0s
Mar 17 07:39:55 vault kernel: sd 0:0:2:0: [sdc] tag#962 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e3 00
Mar 17 07:39:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:39:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:01 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:02 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:03 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:04 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:05 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:06 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:07 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:08 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:09 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:10 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:11 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:21 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:22 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:23 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:40:24 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:41:27 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:41:28 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:41:29 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:41:30 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:32 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:34 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:42:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:43:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:43:37 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:43:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:43:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:43 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:44 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:44:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:45:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:45:45 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:45:46 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:45:47 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:45:48 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:46:50 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:46:51 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:46:52 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:46:53 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:54 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:47:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:48:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:49:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:49:01 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:49:02 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:49:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:49:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:49:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:49:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:49:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:50:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:50:03 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:50:04 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:50:05 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:50:06 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:51:08 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:51:09 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:51:10 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:51:11 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:52:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:53:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:53:18 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:53:19 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:53:20 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:21 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:22 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:23 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:24 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:54:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:55:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:55:26 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:55:27 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:55:28 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:55:29 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:56:30 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:56:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:56:32 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:56:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:37 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:57:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:58:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:58:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:58:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:58:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:44 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:45 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:46 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:47 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 07:59:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:00:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:00:48 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:00:49 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:00:50 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:00:51 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:01:53 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:01:54 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:01:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:01:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:02:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:03:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:02 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:03 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:04 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:05 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:04:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:05:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:05:06 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:05:07 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:05:08 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:05:09 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:06:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:06:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:06:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:06:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:18 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:19 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:07:20 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:08:23 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:08:24 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:08:25 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:08:26 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:28 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:29 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:30 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:09:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:10:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:10:34 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:10:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:10:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:10:37 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:11:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:11:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:11:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:11:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:45 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:46 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:47 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:12:48 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:13:50 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:13:51 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:13:52 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:13:53 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:14:54 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:14:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:14:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:14:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:15:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:15:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:15:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:15:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:15:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:15:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:16:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:16:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:16:01 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:16:02 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:03 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:04 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:05 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:06 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:17:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:18:08 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:18:09 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:18:10 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:18:11 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:19:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:19:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:19:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:19:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:18 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:19 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:20 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:20:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:21:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:21:21 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:21:22 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:21:23 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:21:24 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:26 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:27 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:28 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:22:29 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:23:30 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:23:31 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:23:32 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:23:33 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:24:35 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:24:36 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:24:37 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:24:38 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:39 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:40 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:41 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:42 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:25:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:26:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:26:44 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:26:45 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:26:46 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:26:47 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:48 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:49 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:50 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:27:51 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:28:53 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:28:54 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:28:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:28:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:29:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:29:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:29:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:30:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:30:55 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:30:56 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:30:57 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:30:58 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:30:59 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:31:00 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:31:02 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:31:03 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:31:04 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:31:05 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:06 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:07 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:08 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:09 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:16 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:32:17 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:33:12 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:33:13 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:33:14 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
Mar 17 08:33:15 vault kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
 
Last edited:
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
EasyRhino said:
To chime, in, I don't think it applies to your particular details, but I had a lot of similar sounding device reset messages from truenas with my 9400-16i... but it wasn't the card at all, I had a failing SATA power splitter cable on the drives.
Click to expand...
Yeah thanks for reminding me about the power possibility - I had read that last week and then forgot to try it. I have a second complete power supply now for this chassis so will swap that out and see. I don't think there are any splitters involved though, it's just 5x molex connectors (each on a dedicated cable from the PSU) going into the backplane. But who knows, there could be an issue there.
 
TrevorH

TrevorH

Active Member
Oct 25, 2024
222
96
28
Since the problem has occurred on 2 separate backplanes, do they share anything? Possibly a power cable connection f.e?
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
TrevorH said:
Since the problem has occurred on 2 separate backplanes, do they share anything? Possibly a power cable connection f.e?
Click to expand...
Yeah that's why power is the next thing for me to investigate. I swapped the backplane and reused the same power from before. In my mind it has to be either power or a fundamental issue with the firmware on this backplane ...
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
Update:

I swapped out the entire power supply unit - no change to behavior.

In the meantime, the problem continues. I hadn't mentioned this yet but the other day while moving drives around between backplanes I managed to drop one on my concrete floor ... it bounced twice and has nice dents on the corners. Worked fine for a couple days but then yesterday completely failed. I have a spare so I swapped that in and resilvered it and that was a good stress of the system. The problem occurred on every single 90 minute interval while resilvering was taking place. Like clockwork - looking at timestamps, it's within a second of 5400 seconds every single time. Lasts for 5-10 minutes, then is completely error free for the next 80-85 minutes until the next one. Resilvering is pretty intensive and can constantly for ~24 hours so this helps reinforce that there must be something else happening every 90 minutes that is triggering this. It can't be the HBA overheating or the power supply or a cable. Software must be involved in some way - either backplane firmware, HBA drivers, or TrueNAS itself I guess.

I've started a thread over at the TrueNAS community to see if they have any ideas. None yet though.

At this point I'm hoping that the 9300-8i I ordered solves the problem although I'm not too optimistic about that. I might need to find a new completely different backplane?
 
M

MBastian

Active Member
Jul 17, 2016
339
100
43
Germany
I was curious and googled a bit. The two things that came up are the LSI Controller overheating (unlikely if it works fine on the other backplane) and backplane firmware issues. As far as I can tell you have to contact Gooxi support. These dmesg line might contain the current firmware version.
Code: 
Enclosure         GOOXIBM  4U24SXP 36Sx12G  B134 PQ: 0 ANSI: 6
I find the 90 minutes interval highly suspicious. If your new controller does not fix the issue your might try to boot from an Live Linux USB Image and see if it happens there too.
If everything fails I'd like to advise you to put your production HDDs in the second backplane and put some spares in the misbehaving one to debug further.
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
MBastian said:
I was curious and googled a bit. The two things that came up are the LSI Controller overheating (unlikely if it works fine on the other backplane) and backplane firmware issues. As far as I can tell you have to contact support. These two dmesg lines might contain the current firmware versions.

I find the 90 minutes interval highly suspicious. If your new controller does not fix the issue your might try to boot from an Live Linux USB Image and see if it happens there too.
If everything fails I'd like to advise you to put your production HDDs in the second backplane and put some spares in the misbehaving one to debug further.
Click to expand...
I'm questioning whether the other backplane does work fine - it did for the brief testing I did but at that time I thought this problem happened 100% of the time when booting but I've since learned it sometimes doesn't so it could have been a coincidence. So I might do just that - put the drives back in the back to test that more and I actually have 8x 14TB drives that I was planning to make a second VDEV and add to this pool once this issue is resolved that I could use instead in a new pool on the front backplane for testing. Or like you said, spin up a completely different OS on bare metal and use them instead of my production drives. Good idea. They're totally different drives too - WD whites shucked out of enclosures instead of the 20TB EXOS drives I'm using now so maybe that will show something?

Any recommendations for what distro to use for testing? Is there another out there that has ZFS already in there so it's quick to setup a new pool?
 
M

MBastian

Active Member
Jul 17, 2016
339
100
43
Germany
Darkwing314 said:
Any recommendations for what distro to use for testing? Is there another out there that has ZFS already in there so it's quick to setup a new pool?
Click to expand...
I do not think you need to have ZFS available. Just put some load on a spare drive and see if it stalls out every 90 mins.
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
MBastian said:
I do not think you need to have ZFS available. Just put some load on a spare drive and see if it stalls out every 90 mins.
Click to expand...
Yeah okay, will give that a try. I found it difficult to reproduce before, even doing all sort of continuous performance tests didn't always reproduce it. Resilvering did seem to be very reliable for doing it though. I'll figure something out to try.

I also just realized my motherboard has an older BIOS (only 3 years old though). Maybe worth updating that?
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
Update time. Since I last posted I have tried a number of additional things:

;TLDR: I think it's TrueNAS that is the problem.
  • I moved my production drives back to the rear backplane connected to the 9400-16i HBA. Not a single error from them since I did that which is great.
  • Updated my motherboard bios to the latest - same problem.
  • I added my 8x 14TB drives to the front backplane connected to the 9400-16i HBA, made a new RAIDZ2 VDEV and added it to a new pool. Right away they started having problems. The easiest way now to reproduce the issue is to initiate a big copy from the production drives to the test drives and just wait 90 minutes after boot. Without fail it will have problems every 90 minutes. They transfer at 500MB/s without any errors up until those 90 minute intervals and then they start timing out. Once they recover they will go back to 500MB/s without a single error until the next 90 minute interval.
  • My Supermicro SAS3008-8Le cards showed up so I put one of those in and connected the front backplane to it. They came with relatively old firmware (12.00.02.00) so I updated that to 16.00.14.00 (the latest Supermicro firmware for them) and tested - same problem. Actually worse problem, I'm not sure if it's this card or the drives (WD white drives that were shucked) but the errors are much more severe with all sorts of write errors and drives actually going into critical state.
  • I cross-flashed them to the 9300-8i special firmware (16.00.12.00) - same problem as above.
  • I tried the second one since I had two - same problem as above.
  • I contacted the guy I bought this chassis from again to ask about what HBA he used - turns out he used LSI 9311-8i which is also a SAS3008 card. He had a few laying around (actual ones they used in this exact chassis) and he let me borrow a couple to test.
  • Installed his 9311-8i - turns out they have the -IR firmware on them so possibly these weren't actually used cards but spares that they hadn't flashed to -IT yet (he did tell me to make sure I used -IT firmware because that's what they used). They came with 16.00.10.00-IR which is the latest normal release firmware. I flashed to 16.00.12.00-IT firmware and tested - same problem with data errors all over the place (not just timeout and recovery)!
  • And now things get interesting. At this point I'm confident the HBA is not the problem since I've tried so many different ones. So it has to be either the backplane itself (but I've also tested a second one, so it would be a fundamental issue with it) or it's TrueNAS. Let's try to eliminate TrueNAS.
  • Reconfigured proxmox to only pass the 9400-16i through to TrueNAS so I can keep that running. Spin up a new VM running XigmaNAS, pass through the 9311-8i, create a new pool using the 8x 14TB drives in the front backplane (old test pool was corrupted really bad at this point), initiate copy from TrueNAS production to XigmaNAS test pool. And here we are 16+ hours later, it has been copying non-stop at about 300-400MB/s, and there is not a single error reported in XigmaNAS. ZFS shows the pool is healthy with zero errors anywhere. Not a single timeout (unless XigmaNAS is hiding them from me ... but dmesg doesn't show anything and all zfs commands suggest it is error free).
So what do I conclude? This is a TrueNAS issue!? I still can't explain that or why it doesn't have any issues with the rear backplane. But all signs at this point are pointing to TrueNAS being the source of the issue.

I did find this thread which is about an issue that happens every 90 minutes so I sort of wonder if this is related somehow. In that case it's doing some sort of health check on drives that have spun down and causing them to spin up every 90 minutes. Could it be related? I don't know but this issue was introduced in 25.10 so going to 25.04 might solve it for me (but then I'd worry that it comes back when I eventually upgrade).

Next up on my test plan is to try an older version of TrueNAS (25.04?). Or maybe TrueNAS Core and see if that has the same issue (but I know that's not a very future-proof upgrade path ... )? I might try OpenMediaVault as well just to see what happens and how I like it. I don't need a lot from this - it's 100% file storage and serving those files. I don't need any containers or apps or whatever. Maybe I should just spin up Debian and put ZFS on it ... ?
 
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
Big update. I think I have a probable cause and a way to reliably reproduce the problem on-demand.

TL;DR: it is health check related. I can manually trigger the problem with the command smartctl -x /dev/sdx (where /dev/sdx is any of the drives in my array)

I spent the last few days trying out different distros to see if this is a universal problem or something about TrueNAS.
  • I tried XigmaNAS and was excited to see that I could not reproduce the problem. I was able to write to the array for more than 24 hours continuously (about 30TB written) and not a single timeout or error occurred. This gave me confidence that maybe I was onto something with it being a software issue.
  • Then I decided to jump back to TrueNAS 25.04 (instead of 25.10 that I was originally using). Low and behold, no timeouts or errors there either (another 24 hours and another 30 TB of data written completely error free)
  • Then I upgraded the 25.04 install to 25.10 and I guess I shouldn't be surprised but the issue was back immediately (happened during boot and then every 90 minutes after).
  • Revert that install to 25.04 (change boot environment) and the problem goes away again.
  • So it is something about TrueNAS 25.10 that is triggering the problem.
  • Just for fun, I then spun up OpenMediaVault to give that a try. Also no problems using that initially.
  • Then I was browsing around the interface and found myself in the SMART section and clicked on the button to view extended info for a drive and suddenly the issue happened! Coincidence perhaps? So I tried again and it did it again. Then I paused my file transfer, let the drives go idle, and tried again and this time no error (no problem viewing extended info for any of the drives). Resume file transfer, try again, bam, timeouts and resets.
  • Dug into it a bit and figured out that this 'extended info' page is just calling smartctl -x /dev/sdx. Switch to terminal, call that command, and bam, it happens every time (if file transfer is happening). If I pause file transfer, no problem.
  • If I do smartctl -a /dev/sdx (note the -a instead of -x) then it does not trigger the problem. I have to do -x for it to be triggered.
  • So was is the -x option actually doing? From the printout it seems to be polling some extra info to do with SCT whatever that is.
Below I have a printout of this command when the drive is idle and one when writing. For the writing one I've split it apart where the difference are - the printout pauses at these points, I get a ton of timeout errors in logs for this specific drive, then it continues to the next one where it triggers more timeouts.

smartctl -x /dev/sdb when drive is idle:
Code: 
root@nas04-omv:~# smartctl -x /dev/sdb
smartctl 7.4 2024-10-15 r5620 [x86_64-linux-6.17.13-2-pve] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar (He10/12)
Device Model:     WDC WD140EDGZ-11B2DA2
Serial Number:    2BG2MV9E
LU WWN Device Id: 5 000cca 295c13340
Firmware Version: 85.00A85
User Capacity:    14,000,519,643,136 bytes [14.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/6114
ATA Version is:   ACS-4 published, ANSI INCITS 529-2018
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr  3 16:16:14 2026 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Disabled
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  101) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1532) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   100   001    -    0
  2 Throughput_Performance  --S---   145   145   054    -    60
  3 Spin_Up_Time            POS---   084   084   001    -    320 (Average 320)
  4 Start_Stop_Count        -O--C-   100   100   000    -    37
  5 Reallocated_Sector_Ct   PO--CK   100   100   001    -    0
  7 Seek_Error_Rate         -O-R--   100   100   001    -    0
  8 Seek_Time_Performance   --S---   128   128   020    -    18
  9 Power_On_Hours          -O--C-   095   095   000    -    35668
10 Spin_Retry_Count        -O--C-   100   100   001    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    37
22 Helium_Level            PO---K   100   100   025    -    100
192 Power-Off_Retract_Count -O--CK   100   100   000    -    1556
193 Load_Cycle_Count        -O--C-   100   100   000    -    1556
194 Temperature_Celsius     -O----   030   030   000    -    46 (Min/Max 17/50)
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O---K   100   100   000    -    0
198 Offline_Uncorrectable   ---R--   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O-R--   100   100   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O    255  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0c       GPL     R/O  30001  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x12       GPL     R/O      1  SATA NCQ Non-Data log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x15       GPL     R/W      1  Rebuild Assist log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    256  Current Device Internal Status Data log
0x25       GPL     R/O    256  Saved Device Internal Status Data log
0x2f       GPL     R/O      1  Set Sector Configuration
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xb7           SL  VS       1  Device vendor specific log
0xd8       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     31422         -
# 2  Extended offline    Completed without error       00%     27165         -
# 3  Extended offline    Completed without error       00%     22745         -
# 4  Extended offline    Completed without error       00%     18378         -
# 5  Extended offline    Completed without error       00%     13967         -
# 6  Extended offline    Completed without error       00%      9627         -
# 7  Short offline       Completed without error       00%      9219         -
# 8  Short offline       Completed without error       00%      8475         -
# 9  Short offline       Completed without error       00%      7731         -
#10  Short offline       Completed without error       00%      7011         -
#11  Short offline       Completed without error       00%      6266         -
#12  Short offline       Completed without error       00%      5546         -
#13  Extended offline    Completed without error       00%      5203         -
#14  Short offline       Completed without error       00%      4802         -
#15  Short offline       Completed without error       00%      4058         -
#16  Short offline       Completed without error       00%      3339         -
#17  Short offline       Completed without error       00%      2594         -
#18  Short offline       Completed without error       00%      1875         -
#19  Short offline       Completed without error       00%      1132         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
Device State:                        Active (0)
Current Temperature:                    46 Celsius
Power Cycle Min/Max Temperature:     35/46 Celsius
Lifetime    Min/Max Temperature:     17/50 Celsius
Under/Over Temperature Limit Count:   0/0
SMART Status:                        0xc24f (PASSED)
Minimum supported ERC Time Limit:    65 (6.5 seconds)

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/65 Celsius
Min/Max Temperature Limit:           -40/70 Celsius
Temperature History Size (Index):    128 (125)

Index    Estimated Time   Temperature Celsius
126    2026-04-03 14:09    46  ***************************
...    ..(126 skipped).    ..  ***************************
125    2026-04-03 16:16    46  ***************************

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              37  ---  Lifetime Power-On Resets
0x01  0x010  4           35668  ---  Power-on Hours
0x01  0x018  6    196960491820  ---  Logical Sectors Written
0x01  0x020  6      1297566003  ---  Number of Write Commands
0x01  0x028  6   1138963825107  ---  Logical Sectors Read
0x01  0x030  6      3718455467  ---  Number of Read Commands
0x01  0x038  6    128405821050  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           31101  ---  Spindle Motor Power-on Hours
0x03  0x010  4           31100  ---  Head Flying Hours
0x03  0x018  4            1556  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              46  ---  Current Temperature
0x05  0x010  1              41  N--  Average Short Term Temperature
0x05  0x018  1              35  N--  Average Long Term Temperature
0x05  0x020  1              50  ---  Highest Temperature
0x05  0x028  1              17  ---  Lowest Temperature
0x05  0x030  1              49  N--  Highest Average Short Term Temperature
0x05  0x038  1              21  N--  Lowest Average Short Term Temperature
0x05  0x040  1              47  N--  Highest Average Long Term Temperature
0x05  0x048  1              25  N--  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              65  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             244  ---  Number of Hardware Resets
0x06  0x010  4               0  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           60  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           61  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS
smartctl -x /dev/sdb when drive is writing:
Code: 
root@nas04-omv:~# smartctl -x /dev/sdb
smartctl 7.4 2024-10-15 r5620 [x86_64-linux-6.17.13-2-pve] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar (He10/12)
Device Model:     WDC WD140EDGZ-11B2DA2
Serial Number:    2BG2MV9E
LU WWN Device Id: 5 000cca 295c13340
Firmware Version: 85.00A85
User Capacity:    14,000,519,643,136 bytes [14.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/6114
ATA Version is:   ACS-4 published, ANSI INCITS 529-2018
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr  3 16:17:15 2026 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Disabled
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
First error happens here:
Code: 
Read SCT Status failed: Input/output error
Wt Cache Reorder: Unknown (SCT Feature Control command failed)
Code: 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  101) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1532) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   100   001    -    0
  2 Throughput_Performance  --S---   145   145   054    -    60
  3 Spin_Up_Time            POS---   084   084   001    -    320 (Average 320)
  4 Start_Stop_Count        -O--C-   100   100   000    -    37
  5 Reallocated_Sector_Ct   PO--CK   100   100   001    -    0
  7 Seek_Error_Rate         -O-R--   100   100   001    -    0
  8 Seek_Time_Performance   --S---   128   128   020    -    18
  9 Power_On_Hours          -O--C-   095   095   000    -    35668
10 Spin_Retry_Count        -O--C-   100   100   001    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    37
22 Helium_Level            PO---K   100   100   025    -    100
192 Power-Off_Retract_Count -O--CK   100   100   000    -    1556
193 Load_Cycle_Count        -O--C-   100   100   000    -    1556
194 Temperature_Celsius     -O----   030   030   000    -    46 (Min/Max 17/50)
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O---K   100   100   000    -    0
198 Offline_Uncorrectable   ---R--   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O-R--   100   100   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O    255  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0c       GPL     R/O  30001  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x12       GPL     R/O      1  SATA NCQ Non-Data log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x15       GPL     R/W      1  Rebuild Assist log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    256  Current Device Internal Status Data log
0x25       GPL     R/O    256  Saved Device Internal Status Data log
0x2f       GPL     R/O      1  Set Sector Configuration
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xb7           SL  VS       1  Device vendor specific log
0xd8       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     31422         -
# 2  Extended offline    Completed without error       00%     27165         -
# 3  Extended offline    Completed without error       00%     22745         -
# 4  Extended offline    Completed without error       00%     18378         -
# 5  Extended offline    Completed without error       00%     13967         -
# 6  Extended offline    Completed without error       00%      9627         -
# 7  Short offline       Completed without error       00%      9219         -
# 8  Short offline       Completed without error       00%      8475         -
# 9  Short offline       Completed without error       00%      7731         -
#10  Short offline       Completed without error       00%      7011         -
#11  Short offline       Completed without error       00%      6266         -
#12  Short offline       Completed without error       00%      5546         -
#13  Extended offline    Completed without error       00%      5203         -
#14  Short offline       Completed without error       00%      4802         -
#15  Short offline       Completed without error       00%      4058         -
#16  Short offline       Completed without error       00%      3339         -
#17  Short offline       Completed without error       00%      2594         -
#18  Short offline       Completed without error       00%      1875         -
#19  Short offline       Completed without error       00%      1132         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
Device State:                        Active (0)
Current Temperature:                    46 Celsius
Power Cycle Min/Max Temperature:     35/46 Celsius
Lifetime    Min/Max Temperature:     17/50 Celsius
Under/Over Temperature Limit Count:   0/0
SMART Status:                        0xc24f (PASSED)
Minimum supported ERC Time Limit:    65 (6.5 seconds)
Next error is here:
Code: 
Read SCT Data Table failed: Input/output error
Read SCT Temperature History failed

Read SCT Status failed: Input/output error
SCT (Get) Error Recovery Control command failed
No errors after this point:
Code: 
Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              37  ---  Lifetime Power-On Resets
0x01  0x010  4           35668  ---  Power-on Hours
0x01  0x018  6    196960618876  ---  Logical Sectors Written
0x01  0x020  6      1297566371  ---  Number of Write Commands
0x01  0x028  6   1138963825203  ---  Logical Sectors Read
0x01  0x030  6      3718455474  ---  Number of Read Commands
0x01  0x038  6    128405975600  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           31101  ---  Spindle Motor Power-on Hours
0x03  0x010  4           31100  ---  Head Flying Hours
0x03  0x018  4            1556  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              46  ---  Current Temperature
0x05  0x010  1              41  N--  Average Short Term Temperature
0x05  0x018  1              35  N--  Average Long Term Temperature
0x05  0x020  1              50  ---  Highest Temperature
0x05  0x028  1              17  ---  Lowest Temperature
0x05  0x030  1              49  N--  Highest Average Short Term Temperature
0x05  0x038  1              21  N--  Lowest Average Short Term Temperature
0x05  0x040  1              47  N--  Highest Average Long Term Temperature
0x05  0x048  1              25  N--  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              65  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4             250  ---  Number of Hardware Resets
0x06  0x010  4               0  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           66  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           67  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS
So, if you're still reading this, does this shed any light on the situation? It's clear to me at this point that TrueNAS 25.10 must be calling this command or something similar every 90 minutes as part of it's background drive health checks. 25.04 I guess doesn't do that? To be honest I didn't look at the SMART settings is 25.04 because I wasn't thinking about this at the time - maybe by default it doesn't do the same checks? Or does the more limited -a one which doesn't cause the problem?
 
  • Like
Reactions: StevenDTX
D

Darkwing314

New Member
Oct 30, 2025
29
8
3
After some more testing I've narrowed it down to the specific components of smartctl that are causing the issue. Specifically, it's the -l scttemp and -l scterc parameters that are the issue. The -x parameter includes these whereas the -a one does not.

If my drives are IDLE (not reading/writing to them) then I get something like this and no timeouts:
Code: 
root@nas04-omv:~# smartctl -l scttemp /dev/sdb
smartctl 7.4 2024-10-15 r5620 [x86_64-linux-6.17.13-2-pve] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
Device State:                        Active (0)
Current Temperature:                    46 Celsius
Power Cycle Min/Max Temperature:     35/47 Celsius
Lifetime    Min/Max Temperature:     17/50 Celsius
Under/Over Temperature Limit Count:   0/0
SMART Status:                        0xc24f (PASSED)
Minimum supported ERC Time Limit:    65 (6.5 seconds)

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/65 Celsius
Min/Max Temperature Limit:           -40/70 Celsius
Temperature History Size (Index):    128 (16)

Index    Estimated Time   Temperature Celsius
  17    2026-04-03 18:38    46  ***************************
...    ..(126 skipped).    ..  ***************************
  16    2026-04-03 20:45    46  ***************************

root@nas04-omv:~# smartctl -l scterc /dev/sdb
smartctl 7.4 2024-10-15 r5620 [x86_64-linux-6.17.13-2-pve] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled
Whereas if I'm writing to the drives and do the same command I get this and timeouts with device resets:
Code: 
root@nas04-omv:~# smartctl -l scttemp /dev/sdb
smartctl 7.4 2024-10-15 r5620 [x86_64-linux-6.17.13-2-pve] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
Device State:                        Active (0)
Current Temperature:                    45 Celsius
Power Cycle Min/Max Temperature:     35/47 Celsius
Lifetime    Min/Max Temperature:     17/50 Celsius
Under/Over Temperature Limit Count:   0/0
SMART Status:                        0xc24f (PASSED)
Minimum supported ERC Time Limit:    65 (6.5 seconds)

Read SCT Data Table failed: Input/output error
Read SCT Temperature History failed

root@nas04-omv:~# smartctl -l scterc /dev/sdb
smartctl 7.4 2024-10-15 r5620 [x86_64-linux-6.17.13-2-pve] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

Read SCT Status failed: Input/output error
SCT (Get) Error Recovery Control command failed
It is 100% repeatable and behaves the same for any drive. I can call that function all day if the drives are idle with no problems. As soon as they're being used though it starts timing out the drive and resetting it. Anyone have any tips on where to go from here? Could this still be a backplane limitation?

I can't yet explain why TrueNAS 25.04 doesn't have the issue - maybe it has a different version of smartctl? Or isn't automatically calling it every 90 minutes? That's next on my list to investigate.
 
  • Like
Reactions: StevenDTX
M

MBastian

Active Member
Jul 17, 2016
339
100
43
Germany
You are absolutely sure you cannot reproduce the issue on the rear backplane?

I am suspecting it has something to do with the load the command puts on the PM8044 expander chip. The PM8043 in the backplane is probably more resilient, as it does not have to manage so many drives. I had a look at what changed between TrueNAS 25.4 and 25.10, and it’s a lot. It might be stricter timings for SAS/SATA commands that work on the smaller backplane but fail on the bigger one. If you checked with other Linux distributions with kernel 6.12 or higher and could not reproduce the error, chances are the TrueNAS 25.10 6.12.33 LTS kernel has a regression that was fixed in newer LTS versions. I’d suggest you open a bug ticket and stay with 25.04 for the time being.

Honestly, I am a bit bummed they do not update minor LTS kernel releases regularly. The 6.12.33 release is a year old, and 6.12.80 is current.
 
Last edited:
You must log in or register to reply here.
Top Bottom