Coming from Napp-it SuperStorage Server 6048R-E1CR36L Performance and after 3 years working fine, this is what my data pool currently looks like:
11 Drives faulted nearly at the same time.
The pool degrading starts by 8 drives and the first try with zpool clear tank1 added 3 more drives.
Status above after reboot (cache device is gone).
A test by /var/web-gui/_my/tools/sas3ircu/lsiutil.i386 (napp-it) on one of affected disk finished with no errors!?
But some abnormality at phy counters?
My first step would be to exchange the HBA and the cable to back plane due to several disk failed at the same time.
Does anyone have any experiences in this case?
Code:
# cat /etc/release
OmniOS v11 r151028o
Copyright 2017 OmniTI Computer Consulting, Inc. All rights reserved.
Copyright 2017-2019 OmniOS Community Edition (OmniOSce) Association.
All rights reserved. Use is subject to licence terms.
# zpool status tank1
pool: tank1
state: UNAVAIL
status: One or more devices are faulted in response to persistent errors. There are insufficient replicas for the pool to
continue functioning.
action: Destroy and re-create the pool from a backup source. Manually marking the device
repaired using 'zpool clear' may allow some data to be recovered.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank1 UNAVAIL 0 0 0 insufficient replicas
mirror-0 ONLINE 0 0 0
c2t5000CCA23B0CEF7Dd0 ONLINE 0 0 0
c2t5000CCA23B0D18F9d0 ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
c2t5000CCA23B0CDAE9d0 ONLINE 0 0 0
c2t5000CCA23B0D0E11d0 FAULTED 0 0 0 external device fault
mirror-2 UNAVAIL 0 0 0 insufficient replicas
c2t5000CCA23B0C20C9d0 UNAVAIL 0 0 0 cannot open
c2t5000CCA23B0CA94Dd0 FAULTED 0 0 0 external device fault
mirror-3 ONLINE 0 0 0
c2t5000CCA23B07B701d0 ONLINE 0 0 0
c2t5000CCA23B0C9CD5d0 ONLINE 0 0 0
mirror-4 UNAVAIL 0 0 0 insufficient replicas
c2t5000CCA23B0BE229d0 FAULTED 0 0 0 external device fault
c2t5000CCA23B0C0935d0 UNAVAIL 0 0 0 cannot open
mirror-5 DEGRADED 0 0 0
c2t5000CCA23B0BFDA9d0 ONLINE 0 0 0
c2t5000CCA23B0D25C9d0 UNAVAIL 0 0 0 cannot open
mirror-6 ONLINE 0 0 0
c2t5000CCA23B0B9121d0 ONLINE 0 0 0
c2t5000CCA23B0BFCA1d0 ONLINE 0 0 0
mirror-7 DEGRADED 0 0 0
c2t5000CCA23B0BDA41d0 ONLINE 0 0 0
c2t5000CCA23B0BFBF1d0 FAULTED 0 0 0 external device fault
mirror-8 ONLINE 0 0 0
c2t5000CCA23B0CE5B9d0 ONLINE 0 0 0
c2t5000CCA23B0CE7A9d0 ONLINE 0 0 0
mirror-9 UNAVAIL 0 0 0 insufficient replicas
c2t5000CCA23B0C0901d0 UNAVAIL 0 0 0 cannot open
c2t5000CCA23B0D1BB5d0 FAULTED 0 0 0 external device fault
mirror-10 DEGRADED 0 0 0
c2t5000CCA23B0C00B1d0 FAULTED 0 0 0 external device fault
c2t5000CCA23B0C9BD5d0 ONLINE 0 0 0
mirror-11 DEGRADED 0 0 0
c2t5000CCA23B0A3AE9d0 FAULTED 0 0 0 external device fault
c2t5000CCA23B0CF6D9d0 ONLINE 0 0 0
logs
mirror-12 ONLINE 0 0 0
c1t5002538C401C745Fd0 ONLINE 0 0 0
c1t5002538C401C7462d0 ONLINE 0 0 0
The pool degrading starts by 8 drives and the first try with zpool clear tank1 added 3 more drives.
Status above after reboot (cache device is gone).
A test by /var/web-gui/_my/tools/sas3ircu/lsiutil.i386 (napp-it) on one of affected disk finished with no errors!?
Code:
Select a device: [1-26 or RETURN to quit] 4
1. Alternating, 8-Bit, 00 and FF
2. Alternating, 8-Bit, 55 and AA
3. Incrementing, 8-Bit
4. Walking 1s and 0s, 8-Bit
5. Alternating, 16-Bit, 0000 and FFFF
6. Alternating, 16-Bit, 5555 and AAAA
7. Incrementing, 16-Bit
8. Walking 1s and 0s, 16-Bit
9: Random
10: All B5
11: All 4A
12: Incrementing across iterations (00 through FF)
Select a data pattern: [1-12 or RETURN to quit] 9
Number of blocks per I/O: [1-64 or RETURN to quit] 64
Number of iterations: [1-1000000 or 0 for infinite or RETURN to quit] 10000
Type of I/O: [0=Sequential, 1=Random, default is 0] 1
Stop test on Write, Read, or Compare error? [Yes or No, default is Yes]
Testing started...
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Testing ended...
Code:
1. Inquiry Test
2. WriteBuffer/ReadBuffer/Compare Test
3. Read Test
4. Write/Read/Compare Test
8. Read Capacity / Read Block Limits Test
12. Display phy counters
13. Clear phy counters
14. SATA SMART Read Test
15. SEP (SCSI Enclosure Processor) Test
18. Report LUNs Test
19. Drive firmware download
20. Expander firmware download
21. Read Logical Blocks
99. Reset port
e Enable expert mode in menus
p Enable paged mode
w Enable logging
Diagnostics menu, select an option: [1-99 or e/p/w or 0 to quit] 12
Adapter Phy 0: Link Up, No Errors
Adapter Phy 1: Link Up, No Errors
Adapter Phy 2: Link Up, No Errors
Adapter Phy 3: Link Up, No Errors
Adapter Phy 4: Link Down, No Errors
Adapter Phy 5: Link Down, No Errors
Adapter Phy 6: Link Up, No Errors
Adapter Phy 7: Link Up, No Errors
Expander (Handle 0009) Phy 0: Link Up, No Errors
Expander (Handle 0009) Phy 1: Link Up, No Errors
Expander (Handle 0009) Phy 2: Link Up, No Errors
Expander (Handle 0009) Phy 3: Link Up, No Errors
Expander (Handle 0009) Phy 4: Link Up, No Errors
Expander (Handle 0009) Phy 5: Link Up, No Errors
Expander (Handle 0009) Phy 6: Link Up, No Errors
Expander (Handle 0009) Phy 7: Link Up, No Errors
Expander (Handle 0009) Phy 8: Link Up, No Errors
Expander (Handle 0009) Phy 9: Link Up, No Errors
Expander (Handle 0009) Phy 10: Link Up, No Errors
Expander (Handle 0009) Phy 11: Link Up, No Errors
Expander (Handle 0009) Phy 12: Link Up
Invalid DWord Count 10
Running Disparity Error Count 0
Loss of DWord Synch Count 2
Phy Reset Problem Count 0
Expander (Handle 0009) Phy 13: Link Up
Invalid DWord Count 11
Running Disparity Error Count 0
Loss of DWord Synch Count 2
Phy Reset Problem Count 0
Expander (Handle 0009) Phy 14: Link Up
Invalid DWord Count 12
Running Disparity Error Count 0
Loss of DWord Synch Count 2
Phy Reset Problem Count 0
Expander (Handle 0009) Phy 15: Link Up
Invalid DWord Count 11
Running Disparity Error Count 3
Loss of DWord Synch Count 2
Phy Reset Problem Count 0
Expander (Handle 0009) Phy 16: Link Down, No Errors
Expander (Handle 0009) Phy 17: Link Down, No Errors
Expander (Handle 0009) Phy 18: Link Down, No Errors
Expander (Handle 0009) Phy 19: Link Down, No Errors
Expander (Handle 0009) Phy 20: Link Up, No Errors
Expander (Handle 0009) Phy 21: Link Up, No Errors
Expander (Handle 0009) Phy 22: Link Up, No Errors
Expander (Handle 0009) Phy 23: Link Up, No Errors
Expander (Handle 0009) Phy 24: Link Down, No Errors
Expander (Handle 0009) Phy 25: Link Down, No Errors
Expander (Handle 0009) Phy 26: Link Down, No Errors
Expander (Handle 0009) Phy 27: Link Down, No Errors
Expander (Handle 0009) Phy 28: Link Up, No Errors
Expander (Handle 0009) Phy 29: Link Up, No Errors
Expander (Handle 0009) Phy 30: Link Up, No Errors
Expander (Handle 0009) Phy 31: Link Up, No Errors
Expander (Handle 0009) Phy 32: Link Up, No Errors
Expander (Handle 0009) Phy 33: Link Up, No Errors
Expander (Handle 0009) Phy 34: Link Up, No Errors
Expander (Handle 0009) Phy 35: Link Up, No Errors
Expander (Handle 0009) Phy 36: Link Up, No Errors
Expander (Handle 0009) Phy 37: Link Up, No Errors
Expander (Handle 0009) Phy 38: Link Up, No Errors
Expander (Handle 0009) Phy 39: Link Up, No Errors
Expander (Handle 0009) Phy 40: Link Up, No Errors
Expander (Handle 0009) Phy 41: Link Down, No Errors
Expander (Handle 0009) Phy 42: Link Down, No Errors
Report Phy Error Log failed with result 16
Report Phy Error Log failed with result 16
Report Phy Error Log failed with result 16
Report Phy Error Log failed with result 16
Report Phy Error Log failed with result 16
Report Phy Error Log failed with result 16
Report Phy Error Log failed with result 16
Report Phy Error Log failed with result 16
Expander (Handle 0017) Phy 0: Link Down, No Errors
Expander (Handle 0017) Phy 1: Link Down, No Errors
Expander (Handle 0017) Phy 2: Link Down, No Errors
Expander (Handle 0017) Phy 3: Link Down, No Errors
Expander (Handle 0017) Phy 4: Link Down, No Errors
Expander (Handle 0017) Phy 5: Link Down, No Errors
Expander (Handle 0017) Phy 6: Link Down, No Errors
Expander (Handle 0017) Phy 7: Link Down, No Errors
Expander (Handle 0017) Phy 8: Link Down, No Errors
Expander (Handle 0017) Phy 9: Link Down, No Errors
Expander (Handle 0017) Phy 10: Link Down, No Errors
Expander (Handle 0017) Phy 11: Link Down, No Errors
Expander (Handle 0017) Phy 12: Link Down, No Errors
Expander (Handle 0017) Phy 13: Link Down, No Errors
Expander (Handle 0017) Phy 14: Link Down, No Errors
Expander (Handle 0017) Phy 15: Link Down, No Errors
Expander (Handle 0017) Phy 16: Link Up, No Errors
Expander (Handle 0017) Phy 17: Link Up, No Errors
Expander (Handle 0017) Phy 18: Link Up, No Errors
Expander (Handle 0017) Phy 19: Link Up, No Errors
Expander (Handle 0017) Phy 20: Link Down, No Errors
Expander (Handle 0017) Phy 21: Link Down, No Errors
Expander (Handle 0017) Phy 22: Link Down, No Errors
Expander (Handle 0017) Phy 23: Link Down, No Errors
Expander (Handle 0017) Phy 24: Link Down, No Errors
Expander (Handle 0017) Phy 25: Link Down, No Errors
Expander (Handle 0017) Phy 26: Link Down, No Errors
Expander (Handle 0017) Phy 27: Link Down, No Errors
Expander (Handle 0017) Phy 28: Link Up, No Errors
Expander (Handle 0017) Phy 29: Link Down, No Errors
Expander (Handle 0017) Phy 30: Link Down, No Errors
Does anyone have any experiences in this case?
Last edited: