Hello,
I have a storage server build on top of HPE 380p Gen8 server, supermicro JBOD, ZFS and napp-it. This ZFS storage was used for storing not critical data, and was configured with three raidz3 VDEVs with 15 drives in each one. Today it failed with exactly 15 drives, with 5 failed drives in each vdev. Pool became unavailable. Reboots didn't help. There was a system error on failed LSI controller, which I replaced. Then I export and import pool, and one of three vdev became online. All five drives in that vdev now online. So there are only 10 UNAVAIL disks in the pool right now, with incorrect labeling data status. What else can be done to fix this? All drives are connected and ok, there are no SMART errors (I've checked all failed drives), or any other errors.
Thanks for any advice
This the status of a pool:
pool: Pool1
id: 6639334169153027388
state: UNAVAIL
status: One or more devices are unavailable.
action: The pool cannot be imported due to unavailable devices or data.
config:
Pool1 UNAVAIL insufficient replicas
raidz3-0 UNAVAIL insufficient replicas
c0t50014EE003896026d0 ONLINE
c0t50014EE0038AD80Ed0 ONLINE
c0t50014EE0038AD966d0 ONLINE
c0t50014EE0038AE436d0 ONLINE
c0t50014EE0038AF4B0d0 UNAVAIL incorrect labeling data
c0t50014EE0038B1096d0 ONLINE
c0t50014EE0038B1489d0 ONLINE
c0t50014EE0038B1494d0 UNAVAIL incorrect labeling data
c0t50014EE058DEF533d0 ONLINE
c0t50014EE058DF22ACd0 UNAVAIL incorrect labeling data
c0t50014EE058E00B16d0 UNAVAIL incorrect labeling data
c0t50014EE058E0251Ed0 UNAVAIL incorrect labeling data
c0t50014EE058E04342d0 ONLINE
c0t50014EE058E0435Dd0 ONLINE
c0t50014EE058E04529d0 ONLINE
raidz3-1 ONLINE
c0t50014EE058E04598d0 ONLINE
c0t50014EE0AE35BEFCd0 ONLINE
c0t50014EE0AE35C56Dd0 ONLINE
c0t50014EE0AE35C682d0 ONLINE
c0t50014EE0038B1494d0 ONLINE
c0t50014EE0AE35D550d0 ONLINE
c0t50014EE0AE35E395d0 ONLINE
c0t50014EE6ADE2DD25d0 ONLINE
c0t50014EE0038AF4B0d0 ONLINE
c0t50014EE60332A49Fd0 ONLINE
c0t50014EE60337A401d0 ONLINE
c0t50014EE60337B88Cd0 ONLINE
c0t50014EE603384593d0 ONLINE
c0t50014EE603384947d0 ONLINE
c0t50014EE6033864FFd0 ONLINE
raidz3-2 UNAVAIL insufficient replicas
c0t50014EE6033865C5d0 ONLINE
c0t50014EE6588D024Fd0 ONLINE
c0t50014EE6588D0CFBd0 UNAVAIL incorrect labeling data
c0t50014EE6588D0D2Fd0 ONLINE
c0t50014EE6588D10F6d0 ONLINE
c0t50014EE6588D1120d0 ONLINE
c0t50014EE6588DA81Bd0 UNAVAIL incorrect labeling data
c0t50014EE6ADE21C50d0 ONLINE
c0t50014EE6ADE220C6d0 UNAVAIL incorrect labeling data
c0t50014EE6ADE22BC3d0 ONLINE
c0t50014EE6ADE2C915d0 ONLINE
c0t50014EE6ADE2DCFDd0 ONLINE
c0t50014EE6ADE2DD25d0 UNAVAIL incorrect labeling data
c0t50014EE6ADE2E2EAd0 UNAVAIL incorrect labeling data
c0t50014EE6ADE2E317d0 ONLINE
device details:
c0t50014EE0038AF4B0d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE0038B1494d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE058DF22ACd0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE058E00B16d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE058E0251Ed0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6588D0CFBd0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6588DA81Bd0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6ADE220C6d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6ADE2DD25d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6ADE2E2EAd0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
I have a storage server build on top of HPE 380p Gen8 server, supermicro JBOD, ZFS and napp-it. This ZFS storage was used for storing not critical data, and was configured with three raidz3 VDEVs with 15 drives in each one. Today it failed with exactly 15 drives, with 5 failed drives in each vdev. Pool became unavailable. Reboots didn't help. There was a system error on failed LSI controller, which I replaced. Then I export and import pool, and one of three vdev became online. All five drives in that vdev now online. So there are only 10 UNAVAIL disks in the pool right now, with incorrect labeling data status. What else can be done to fix this? All drives are connected and ok, there are no SMART errors (I've checked all failed drives), or any other errors.
Thanks for any advice
This the status of a pool:
pool: Pool1
id: 6639334169153027388
state: UNAVAIL
status: One or more devices are unavailable.
action: The pool cannot be imported due to unavailable devices or data.
config:
Pool1 UNAVAIL insufficient replicas
raidz3-0 UNAVAIL insufficient replicas
c0t50014EE003896026d0 ONLINE
c0t50014EE0038AD80Ed0 ONLINE
c0t50014EE0038AD966d0 ONLINE
c0t50014EE0038AE436d0 ONLINE
c0t50014EE0038AF4B0d0 UNAVAIL incorrect labeling data
c0t50014EE0038B1096d0 ONLINE
c0t50014EE0038B1489d0 ONLINE
c0t50014EE0038B1494d0 UNAVAIL incorrect labeling data
c0t50014EE058DEF533d0 ONLINE
c0t50014EE058DF22ACd0 UNAVAIL incorrect labeling data
c0t50014EE058E00B16d0 UNAVAIL incorrect labeling data
c0t50014EE058E0251Ed0 UNAVAIL incorrect labeling data
c0t50014EE058E04342d0 ONLINE
c0t50014EE058E0435Dd0 ONLINE
c0t50014EE058E04529d0 ONLINE
raidz3-1 ONLINE
c0t50014EE058E04598d0 ONLINE
c0t50014EE0AE35BEFCd0 ONLINE
c0t50014EE0AE35C56Dd0 ONLINE
c0t50014EE0AE35C682d0 ONLINE
c0t50014EE0038B1494d0 ONLINE
c0t50014EE0AE35D550d0 ONLINE
c0t50014EE0AE35E395d0 ONLINE
c0t50014EE6ADE2DD25d0 ONLINE
c0t50014EE0038AF4B0d0 ONLINE
c0t50014EE60332A49Fd0 ONLINE
c0t50014EE60337A401d0 ONLINE
c0t50014EE60337B88Cd0 ONLINE
c0t50014EE603384593d0 ONLINE
c0t50014EE603384947d0 ONLINE
c0t50014EE6033864FFd0 ONLINE
raidz3-2 UNAVAIL insufficient replicas
c0t50014EE6033865C5d0 ONLINE
c0t50014EE6588D024Fd0 ONLINE
c0t50014EE6588D0CFBd0 UNAVAIL incorrect labeling data
c0t50014EE6588D0D2Fd0 ONLINE
c0t50014EE6588D10F6d0 ONLINE
c0t50014EE6588D1120d0 ONLINE
c0t50014EE6588DA81Bd0 UNAVAIL incorrect labeling data
c0t50014EE6ADE21C50d0 ONLINE
c0t50014EE6ADE220C6d0 UNAVAIL incorrect labeling data
c0t50014EE6ADE22BC3d0 ONLINE
c0t50014EE6ADE2C915d0 ONLINE
c0t50014EE6ADE2DCFDd0 ONLINE
c0t50014EE6ADE2DD25d0 UNAVAIL incorrect labeling data
c0t50014EE6ADE2E2EAd0 UNAVAIL incorrect labeling data
c0t50014EE6ADE2E317d0 ONLINE
device details:
c0t50014EE0038AF4B0d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE0038B1494d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE058DF22ACd0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE058E00B16d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE058E0251Ed0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6588D0CFBd0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6588DA81Bd0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6ADE220C6d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6ADE2DD25d0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.
c0t50014EE6ADE2E2EAd0 UNAVAIL incorrect labeling data
status: ZFS detected errors on this device.
The device has bad label or disk contents.