This may be lengthy.
Short story we had a storm while i was at work I called wife and walked her through powering down my servers over the phone when she got to my openindiana/napp-it server under shutdown she chose the Red option "Emergency Shutdown"! when i got home to power on server and checked stuff it shown c5t5d0 as removed with no stats.
I have tried zpool remove , zpool replace, zpool detach and a few other commands inclusing zpool clear and zpool online.
Zpool shows the drive removed Disks shows that drive isn't even there. The Dell Perc5/i card shown Foreign so I replaced the drive and fired it up. and it still shows removed. tried all the commands and these are the results.
I am running a Raidz-2 with mirrors log drives and 3 cache drives
Pool VER RAW SIZE/ USABLE ALLOC RES FRES AVAIL zfs [df -h/df -H] DEDUP FAILM EXPAND REPLACE ALT GUID HEALTH SYNC ENCRYPT ACTION
rpool - 149G/ 147.1GB 17.8G - - 121G [121G /130G] 1.00x wait off off - 17485873680110936703 ONLINE standard n.a. clear errors
tank01 - 10.9T/ 7.8TB 7.28T - 727G 2.28T [2.3T /2.6T] 1.00x wait on on - 14839772986846248536 DEGRADED standard n.a. clear errors
update 02:54 25 s
Info: RAW poolsize does not count redundancy, usable/available size is from zfs list, df -h displays size as a power of 1024 wheras df -H displays as a power of 1000
------------
zpool status
pool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0 in 0h12m with 0 errors on Sun Aug 25 06:12:26 2013
config:
NAME STATE READ WRITE CKSUM CAP Product
rpool ONLINE 0 0 0
c4t5d0s0 ONLINE 0 0 0 160 GB ST9160823AS
errors: No known data errors
pool: tank01
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0 in 10h16m with 0 errors on Mon Aug 26 16:16:50 2013
config:
NAME STATE READ WRITE CKSUM CAP Product
tank01 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c5t0d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t1d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t2d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t3d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t4d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t5d0 REMOVED 0 0 0
logs
mirror-1 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t4d0 ONLINE 0 0 0 32 GB Patriot Torqx 2
cache
c4t0d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t1d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t2d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
errors: No known data errors
---
zdb -C pool, please wait..
Pool Version Pool GUID Vdev Ashift Asize Vdev GUID Disk Disk-GUID Cap Product/ Phys_Path/ Dev_Id/ Sn
rpool 5000 17485873680110936703 vdevs: 1
vdev 1: disk 9 160.00 GB 13221441911740557431
c4t5d0 ST9160823AS
/pci@0,0/pci1462,7756@1f,2/disk@5,0:a
id1,sd@SATA_____ST9160823AS_________________5NK1J263/a
5NK1J263
tank01 5000 14839772986846248536 vdevs: 2
vdev 1: raidz2 9 12.00 TB 367325143694507204
c5t0d0 462117676879670297 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@0,0:a
id1,sd@n60019b90b0b6320016d5a861775f5b5d/a
005D5B5F7761A8D
c5t1d0 18230133605271295462 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@1,0:a
id1,sd@n60019b90b0b6320016d5a8cdf234d216/a
0016D234F2CDA8D
c5t2d0 9409131566896620802 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@2,0:a
id1,sd@n60019b90b0b6320016d5a8ffb356c408/a
0008C456B3FFA8D
c5t3d0 11639681875289391936 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@3,0:a
id1,sd@n60019b90b0b6320016d5a96732a9d169/a
0069D1A93267A9D
c5t4d0 13830573853347970220 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@4,0:a
id1,sd@n60019b90b0b6320017f90ad7ff2f2708/a
0008272FFFD70AF
c5t5d0 6119548367555395378
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@5,0:a
id1,sd@n60019b90b0b6320017f90b467572599c/a
vdev 2: mirror 9 32.00 GB 2278018801403795219
c4t3d0 14760996154911623107 OCZ VERTEX-TURBO
/pci@0,0/pci1462,7756@1f,2/disk@3,0:a
id1,sd@SATA_____OCZ_VERTEX-TURBOV1YK9NG6Q57876F2FX6R/a
V1YK9NG6Q57876F
c4t4d0 3097851024929752082 Patriot Torqx 2
/pci@0,0/pci1462,7756@1f,2/disk@4,0:a
id1,sd@SATA_____Patriot_Torqx_2_879E07210A0800021980/a
879E07210A08000
------
id part identify stat diskcap partcap partcap2 error vendor product sn
c4t0d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO T1CEJT5B5631FC2
c4t1d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO 84Q9G7T0QC9UO6Y
c4t2d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO WZOBTL883ECWJT5
c4t3d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO V1YK9NG6Q57876F
c4t4d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA Patriot Torqx 2 879E07210A08000
c4t5d0 1 via dd ok 160 GB 160 GB 149.1 GiB S:0 H:0 T:0 ATA ST9160823AS 5NK1J263
c5t0d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 005D5B5F7761A8D
c5t1d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0016D234F2CDA8D
c5t2d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0008C456B3FFA8D
c5t3d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0069D1A93267A9D
c5t4d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0008272FFFD70AF
*** notice no c5t5d0 ****
List of all partitions:
id part stat pool ptable pstart pend diskcap partcap ptype pfs pflags lsec psec vendor product sn
c4t0d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO T1CEJT5B5631FC2
c4t1d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO 84Q9G7T0QC9UO6Y
c4t2d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO WZOBTL883ECWJT5
c4t3d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO V1YK9NG6Q57876F
c4t4d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA Patriot Torqx 2 879E07210A08000
c4t5d0 1 ok rpool msdos - - 160 GB 160 GB - - - ATA ST9160823AS 5NK1J263
c5t0d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 005D5B5F7761A8D
c5t1d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0016D234F2CDA8D
c5t2d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0008C456B3FFA8D
c5t3d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0069D1A93267A9D
c5t4d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0008272FFFD70AF
System Info
ardware via prtdiag, Memory via prtconf:
Memory size: 16258 Megabytes
System Configuration: MSI MS-7756
BIOS Configuration: American Megatrends Inc. V1.7 07/09/2012
==== Processor Sockets ====================================
Version Location Tag
-------------------------------- --------------------------
Intel(R) Celeron(R) CPU G530 @ 2.40GHz SOCKET 0
==== Memory Device Sockets ================================
Type Status Set Device Locator Bank Locator
----------- ------ --- ------------------- ----------------
Unknown in use 0 ChannelA-DIMM0 BANK 0
Unknown in use 0 ChannelA-DIMM1 BANK 1
Unknown in use 0 ChannelB-DIMM0 BANK 2
Unknown in use 0 ChannelB-DIMM1 BANK 3
==== On-Board Devices =====================================
==== Upgradeable Slots ====================================
ID Status Type Description
--- --------- ---------------- ----------------------------
0 available PCI Express J6B2
1 in use PCI Express J6B1
2 in use PCI Express J6D1
3 available PCI Express J7B1
4 available PCI Express J8B4
5 available PCI J8D1
6 available PCI J8B3
Net Overview
DEVICE LINK MEDIA STATE SPEED DUPLEX DHCP IP NETMASK MAC MTU
bge0 bge0 Ethernet up 1000 full static -- - no such interface -
bge1 bge1 Ethernet up 1000 full static -- - no such interface -
rge0 rge0 Ethernet up 1000 full static -- - no such interface -
If you have a live version with GUI, change ip-settings on your server with the locally installed browser and url Activity Stream - ServeTheHome.com Forums
You should also add an entry for your ip adresses in hosts (/etc/hosts) like: 192.168.1.2 fs01
Network service-status, ifconfig -a:
online 8:10:42 svc:/network/physical:default
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
aggr1: flags=1000843 mtu 1500 index 2
inet 192.168.2.49 netmask ffffff00 broadcast 192.168.2.255
ether xx:xx:xx:xx:xx:
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
3 Nic Aggregated link nice throughput on a Dell managed 24 port switch.
any advice would be appreciated.
File Server is used for usual file serving and media streaming .... I also Hold my VMWare Guests on this and do some system backups to the SAN I am hoping the vm's take advantage of the SSD I also have 3 SSD as host_cache on my esxi server.
Short story we had a storm while i was at work I called wife and walked her through powering down my servers over the phone when she got to my openindiana/napp-it server under shutdown she chose the Red option "Emergency Shutdown"! when i got home to power on server and checked stuff it shown c5t5d0 as removed with no stats.
I have tried zpool remove , zpool replace, zpool detach and a few other commands inclusing zpool clear and zpool online.
Zpool shows the drive removed Disks shows that drive isn't even there. The Dell Perc5/i card shown Foreign so I replaced the drive and fired it up. and it still shows removed. tried all the commands and these are the results.
I am running a Raidz-2 with mirrors log drives and 3 cache drives
Pool VER RAW SIZE/ USABLE ALLOC RES FRES AVAIL zfs [df -h/df -H] DEDUP FAILM EXPAND REPLACE ALT GUID HEALTH SYNC ENCRYPT ACTION
rpool - 149G/ 147.1GB 17.8G - - 121G [121G /130G] 1.00x wait off off - 17485873680110936703 ONLINE standard n.a. clear errors
tank01 - 10.9T/ 7.8TB 7.28T - 727G 2.28T [2.3T /2.6T] 1.00x wait on on - 14839772986846248536 DEGRADED standard n.a. clear errors
update 02:54 25 s
Info: RAW poolsize does not count redundancy, usable/available size is from zfs list, df -h displays size as a power of 1024 wheras df -H displays as a power of 1000
------------
zpool status
pool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0 in 0h12m with 0 errors on Sun Aug 25 06:12:26 2013
config:
NAME STATE READ WRITE CKSUM CAP Product
rpool ONLINE 0 0 0
c4t5d0s0 ONLINE 0 0 0 160 GB ST9160823AS
errors: No known data errors
pool: tank01
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0 in 10h16m with 0 errors on Mon Aug 26 16:16:50 2013
config:
NAME STATE READ WRITE CKSUM CAP Product
tank01 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c5t0d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t1d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t2d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t3d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t4d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t5d0 REMOVED 0 0 0
logs
mirror-1 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t4d0 ONLINE 0 0 0 32 GB Patriot Torqx 2
cache
c4t0d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t1d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t2d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
errors: No known data errors
---
zdb -C pool, please wait..
Pool Version Pool GUID Vdev Ashift Asize Vdev GUID Disk Disk-GUID Cap Product/ Phys_Path/ Dev_Id/ Sn
rpool 5000 17485873680110936703 vdevs: 1
vdev 1: disk 9 160.00 GB 13221441911740557431
c4t5d0 ST9160823AS
/pci@0,0/pci1462,7756@1f,2/disk@5,0:a
id1,sd@SATA_____ST9160823AS_________________5NK1J263/a
5NK1J263
tank01 5000 14839772986846248536 vdevs: 2
vdev 1: raidz2 9 12.00 TB 367325143694507204
c5t0d0 462117676879670297 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@0,0:a
id1,sd@n60019b90b0b6320016d5a861775f5b5d/a
005D5B5F7761A8D
c5t1d0 18230133605271295462 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@1,0:a
id1,sd@n60019b90b0b6320016d5a8cdf234d216/a
0016D234F2CDA8D
c5t2d0 9409131566896620802 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@2,0:a
id1,sd@n60019b90b0b6320016d5a8ffb356c408/a
0008C456B3FFA8D
c5t3d0 11639681875289391936 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@3,0:a
id1,sd@n60019b90b0b6320016d5a96732a9d169/a
0069D1A93267A9D
c5t4d0 13830573853347970220 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@4,0:a
id1,sd@n60019b90b0b6320017f90ad7ff2f2708/a
0008272FFFD70AF
c5t5d0 6119548367555395378
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@5,0:a
id1,sd@n60019b90b0b6320017f90b467572599c/a
vdev 2: mirror 9 32.00 GB 2278018801403795219
c4t3d0 14760996154911623107 OCZ VERTEX-TURBO
/pci@0,0/pci1462,7756@1f,2/disk@3,0:a
id1,sd@SATA_____OCZ_VERTEX-TURBOV1YK9NG6Q57876F2FX6R/a
V1YK9NG6Q57876F
c4t4d0 3097851024929752082 Patriot Torqx 2
/pci@0,0/pci1462,7756@1f,2/disk@4,0:a
id1,sd@SATA_____Patriot_Torqx_2_879E07210A0800021980/a
879E07210A08000
------
id part identify stat diskcap partcap partcap2 error vendor product sn
c4t0d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO T1CEJT5B5631FC2
c4t1d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO 84Q9G7T0QC9UO6Y
c4t2d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO WZOBTL883ECWJT5
c4t3d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO V1YK9NG6Q57876F
c4t4d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA Patriot Torqx 2 879E07210A08000
c4t5d0 1 via dd ok 160 GB 160 GB 149.1 GiB S:0 H:0 T:0 ATA ST9160823AS 5NK1J263
c5t0d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 005D5B5F7761A8D
c5t1d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0016D234F2CDA8D
c5t2d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0008C456B3FFA8D
c5t3d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0069D1A93267A9D
c5t4d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0008272FFFD70AF
*** notice no c5t5d0 ****
List of all partitions:
id part stat pool ptable pstart pend diskcap partcap ptype pfs pflags lsec psec vendor product sn
c4t0d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO T1CEJT5B5631FC2
c4t1d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO 84Q9G7T0QC9UO6Y
c4t2d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO WZOBTL883ECWJT5
c4t3d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO V1YK9NG6Q57876F
c4t4d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA Patriot Torqx 2 879E07210A08000
c4t5d0 1 ok rpool msdos - - 160 GB 160 GB - - - ATA ST9160823AS 5NK1J263
c5t0d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 005D5B5F7761A8D
c5t1d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0016D234F2CDA8D
c5t2d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0008C456B3FFA8D
c5t3d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0069D1A93267A9D
c5t4d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0008272FFFD70AF
System Info
ardware via prtdiag, Memory via prtconf:
Memory size: 16258 Megabytes
System Configuration: MSI MS-7756
BIOS Configuration: American Megatrends Inc. V1.7 07/09/2012
==== Processor Sockets ====================================
Version Location Tag
-------------------------------- --------------------------
Intel(R) Celeron(R) CPU G530 @ 2.40GHz SOCKET 0
==== Memory Device Sockets ================================
Type Status Set Device Locator Bank Locator
----------- ------ --- ------------------- ----------------
Unknown in use 0 ChannelA-DIMM0 BANK 0
Unknown in use 0 ChannelA-DIMM1 BANK 1
Unknown in use 0 ChannelB-DIMM0 BANK 2
Unknown in use 0 ChannelB-DIMM1 BANK 3
==== On-Board Devices =====================================
==== Upgradeable Slots ====================================
ID Status Type Description
--- --------- ---------------- ----------------------------
0 available PCI Express J6B2
1 in use PCI Express J6B1
2 in use PCI Express J6D1
3 available PCI Express J7B1
4 available PCI Express J8B4
5 available PCI J8D1
6 available PCI J8B3
Net Overview
DEVICE LINK MEDIA STATE SPEED DUPLEX DHCP IP NETMASK MAC MTU
bge0 bge0 Ethernet up 1000 full static -- - no such interface -
bge1 bge1 Ethernet up 1000 full static -- - no such interface -
rge0 rge0 Ethernet up 1000 full static -- - no such interface -
If you have a live version with GUI, change ip-settings on your server with the locally installed browser and url Activity Stream - ServeTheHome.com Forums
You should also add an entry for your ip adresses in hosts (/etc/hosts) like: 192.168.1.2 fs01
Network service-status, ifconfig -a:
online 8:10:42 svc:/network/physical:default
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
aggr1: flags=1000843 mtu 1500 index 2
inet 192.168.2.49 netmask ffffff00 broadcast 192.168.2.255
ether xx:xx:xx:xx:xx:
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
3 Nic Aggregated link nice throughput on a Dell managed 24 port switch.
any advice would be appreciated.
File Server is used for usual file serving and media streaming .... I also Hold my VMWare Guests on this and do some system backups to the SAN I am hoping the vm's take advantage of the SSD I also have 3 SSD as host_cache on my esxi server.
Last edited: