zpool c5t5d0 showing drive removed !

[2g33k4u]

This may be lengthy.

Short story we had a storm while i was at work I called wife and walked her through powering down my servers over the phone when she got to my openindiana/napp-it server under shutdown she chose the Red option "Emergency Shutdown"! when i got home to power on server and checked stuff it shown c5t5d0 as removed with no stats.

I have tried zpool remove , zpool replace, zpool detach and a few other commands inclusing zpool clear and zpool online.

Zpool shows the drive removed Disks shows that drive isn't even there. The Dell Perc5/i card shown Foreign so I replaced the drive and fired it up. and it still shows removed. tried all the commands and these are the results.

I am running a Raidz-2 with mirrors log drives and 3 cache drives

Pool VER RAW SIZE/ USABLE ALLOC RES FRES AVAIL zfs [df -h/df -H] DEDUP FAILM EXPAND REPLACE ALT GUID HEALTH SYNC ENCRYPT ACTION
rpool - 149G/ 147.1GB 17.8G - - 121G [121G /130G] 1.00x wait off off - 17485873680110936703 ONLINE standard n.a. clear errors
tank01 - 10.9T/ 7.8TB 7.28T - 727G 2.28T [2.3T /2.6T] 1.00x wait on on - 14839772986846248536 DEGRADED standard n.a. clear errors
update 02:54 25 s
Info: RAW poolsize does not count redundancy, usable/available size is from zfs list, df -h displays size as a power of 1024 wheras df -H displays as a power of 1000

------------

zpool status
pool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0 in 0h12m with 0 errors on Sun Aug 25 06:12:26 2013
config:

NAME STATE READ WRITE CKSUM CAP Product
rpool ONLINE 0 0 0
c4t5d0s0 ONLINE 0 0 0 160 GB ST9160823AS

errors: No known data errors

pool: tank01
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0 in 10h16m with 0 errors on Mon Aug 26 16:16:50 2013
config:

NAME STATE READ WRITE CKSUM CAP Product
tank01 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c5t0d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t1d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t2d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t3d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t4d0 ONLINE 0 0 0 2 TB PERC 5/i
c5t5d0 REMOVED 0 0 0
logs
mirror-1 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t4d0 ONLINE 0 0 0 32 GB Patriot Torqx 2
cache
c4t0d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t1d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO
c4t2d0 ONLINE 0 0 0 32 GB OCZ VERTEX-TURBO

errors: No known data errors

---

zdb -C pool, please wait..
Pool Version Pool GUID Vdev Ashift Asize Vdev GUID Disk Disk-GUID Cap Product/ Phys_Path/ Dev_Id/ Sn

rpool 5000 17485873680110936703 vdevs: 1
vdev 1: disk 9 160.00 GB 13221441911740557431
c4t5d0 ST9160823AS
/pci@0,0/pci1462,7756@1f,2/disk@5,0:a
id1,sd@SATA_____ST9160823AS_________________5NK1J263/a
5NK1J263

tank01 5000 14839772986846248536 vdevs: 2
vdev 1: raidz2 9 12.00 TB 367325143694507204
c5t0d0 462117676879670297 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@0,0:a
id1,sd@n60019b90b0b6320016d5a861775f5b5d/a
005D5B5F7761A8D
c5t1d0 18230133605271295462 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@1,0:a
id1,sd@n60019b90b0b6320016d5a8cdf234d216/a
0016D234F2CDA8D
c5t2d0 9409131566896620802 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@2,0:a
id1,sd@n60019b90b0b6320016d5a8ffb356c408/a
0008C456B3FFA8D
c5t3d0 11639681875289391936 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@3,0:a
id1,sd@n60019b90b0b6320016d5a96732a9d169/a
0069D1A93267A9D
c5t4d0 13830573853347970220 PERC 5/i
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@4,0:a
id1,sd@n60019b90b0b6320017f90ad7ff2f2708/a
0008272FFFD70AF
c5t5d0 6119548367555395378
/pci@0,0/pci8086,101@1/pci8086,370@0/pci1028,1f03@e/sd@5,0:a
id1,sd@n60019b90b0b6320017f90b467572599c/a

vdev 2: mirror 9 32.00 GB 2278018801403795219
c4t3d0 14760996154911623107 OCZ VERTEX-TURBO
/pci@0,0/pci1462,7756@1f,2/disk@3,0:a
id1,sd@SATA_____OCZ_VERTEX-TURBOV1YK9NG6Q57876F2FX6R/a
V1YK9NG6Q57876F
c4t4d0 3097851024929752082 Patriot Torqx 2
/pci@0,0/pci1462,7756@1f,2/disk@4,0:a
id1,sd@SATA_____Patriot_Torqx_2_879E07210A0800021980/a
879E07210A08000

------


id part identify stat diskcap partcap partcap2 error vendor product sn
c4t0d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO T1CEJT5B5631FC2
c4t1d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO 84Q9G7T0QC9UO6Y
c4t2d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO WZOBTL883ECWJT5
c4t3d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA OCZ VERTEX-TURBO V1YK9NG6Q57876F
c4t4d0 1 via dd ok 32.0 GB 32 GB 29.8 GiB S:0 H:0 T:0 ATA Patriot Torqx 2 879E07210A08000
c4t5d0 1 via dd ok 160 GB 160 GB 149.1 GiB S:0 H:0 T:0 ATA ST9160823AS 5NK1J263
c5t0d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 005D5B5F7761A8D
c5t1d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0016D234F2CDA8D
c5t2d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0008C456B3FFA8D
c5t3d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0069D1A93267A9D
c5t4d0 1 via dd ok 2000 GB 2 TB 1.8 TiB S:0 H:0 T:0 DELL PERC 5/i 0008272FFFD70AF

*** notice no c5t5d0 ****

List of all partitions:
id part stat pool ptable pstart pend diskcap partcap ptype pfs pflags lsec psec vendor product sn
c4t0d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO T1CEJT5B5631FC2
c4t1d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO 84Q9G7T0QC9UO6Y
c4t2d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO WZOBTL883ECWJT5
c4t3d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA OCZ VERTEX-TURBO V1YK9NG6Q57876F
c4t4d0 1 ok tank01 gpt - - 32.0 GB 32 GB - zfs - ATA Patriot Torqx 2 879E07210A08000
c4t5d0 1 ok rpool msdos - - 160 GB 160 GB - - - ATA ST9160823AS 5NK1J263
c5t0d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 005D5B5F7761A8D
c5t1d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0016D234F2CDA8D
c5t2d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0008C456B3FFA8D
c5t3d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0069D1A93267A9D
c5t4d0 1 ok tank01 gpt - - 2000 GB 2 TB - zfs - DELL PERC 5/i 0008272FFFD70AF

System Info

ardware via prtdiag, Memory via prtconf:

Memory size: 16258 Megabytes

System Configuration: MSI MS-7756
BIOS Configuration: American Megatrends Inc. V1.7 07/09/2012

==== Processor Sockets ====================================

Version Location Tag
-------------------------------- --------------------------
Intel(R) Celeron(R) CPU G530 @ 2.40GHz SOCKET 0

==== Memory Device Sockets ================================

Type Status Set Device Locator Bank Locator
----------- ------ --- ------------------- ----------------
Unknown in use 0 ChannelA-DIMM0 BANK 0
Unknown in use 0 ChannelA-DIMM1 BANK 1
Unknown in use 0 ChannelB-DIMM0 BANK 2
Unknown in use 0 ChannelB-DIMM1 BANK 3

==== On-Board Devices =====================================

==== Upgradeable Slots ====================================

ID Status Type Description
--- --------- ---------------- ----------------------------
0 available PCI Express J6B2
1 in use PCI Express J6B1
2 in use PCI Express J6D1
3 available PCI Express J7B1
4 available PCI Express J8B4
5 available PCI J8D1
6 available PCI J8B3

Net Overview

DEVICE LINK MEDIA STATE SPEED DUPLEX DHCP IP NETMASK MAC MTU

bge0 bge0 Ethernet up 1000 full static -- - no such interface -
bge1 bge1 Ethernet up 1000 full static -- - no such interface -
rge0 rge0 Ethernet up 1000 full static -- - no such interface -

If you have a live version with GUI, change ip-settings on your server with the locally installed browser and url Activity Stream - ServeTheHome.com Forums
You should also add an entry for your ip adresses in hosts (/etc/hosts) like: 192.168.1.2 fs01

Network service-status, ifconfig -a:
online 8:10:42 svc:/network/physical:default

lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
aggr1: flags=1000843 mtu 1500 index 2
inet 192.168.2.49 netmask ffffff00 broadcast 192.168.2.255
ether xx:xx:xx:xx:xx:
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128

3 Nic Aggregated link nice throughput on a Dell managed 24 port switch.

any advice would be appreciated.

File Server is used for usual file serving and media streaming .... I also Hold my VMWare Guests on this and do some system backups to the SAN I am hoping the vm's take advantage of the SSD I also have 3 SSD as host_cache on my esxi server.

 

[2g33k4u]

Decided to remove the mirrored SSD ZiL and add them to the Cache so now 5 SSD for cache and should make the c4t3d0 and c4t4d0 last much longer. I am jumping system up to 32gb this weekend so shouldn't really need the zil

 

[gea]

Emergency shutdown down does not wait for clients to finish writes. It is similar to just power off. This is not a problem for ZFS.

I suppose you have a simple cable or disk failure.
I would check cabling and then the disk with a manufacturers tool.

 

[2g33k4u]

Actually I fixed the problem. Somehow HBA got borked and marked a drive as foreign. after a bit of research I discovered the option to import from the hba bios and it put the drive back to normal. I boot the system up and OI automatically started reslivering life is good again.

 

[mrkrad]

It's this stuff that keeps scaring me away from ZFS setups.

My san is limited but the lefthand was stupid easy to manage and understand every step with best practice guidance (aka your doing it wrong tips!). it seems the ui for most ZFS leaves much to be desired.

If you need more than 1 hour to learn to use a SAN/NAS - poor ui design imo

 

[nle]

Then again ZFS saves you time and time again. So I guess it is worth the effort to learn about it. Napp-it (made by @gea) also makes it quite easy to set up everything out of the box –*he has also written a few guides on the most common setups.

 

[Aluminum]

hashed your data lately? yeah, about that...

It's this stuff that keeps scaring me away from ZFS setups.

My san is limited but the lefthand was stupid easy to manage and understand every step with best practice guidance (aka your doing it wrong tips!). it seems the ui for most ZFS leaves much to be desired.

If you need more than 1 hour to learn to use a SAN/NAS - poor ui design imo

ZFS is one of the few things that actually gives a damn about data integrity and can prove it. None of the hardware raid cards do it right, nor anything that runs on windows.

It also has no UI, just two commands with lots of possible arguments. Nothing worth doing right only takes an hour.

 

[gea]

It's this stuff that keeps scaring me away from ZFS setups.

My san is limited but the lefthand was stupid easy to manage and understand every step with best practice guidance (aka your doing it wrong tips!). it seems the ui for most ZFS leaves much to be desired.

If you need more than 1 hour to learn to use a SAN/NAS - poor ui design imo

Dell SAS Controllers are not the first choice for ZFS because quite often you cannot reflash them to pure HBAs without Raid (IT mode) so sometimes you have some extra hazzles with them that you do not have with a LSI 9211/IT mode or a stock LSI 9207.

But the point is:
I doubt that you can recommend any other storage system with a similar data security, data expandability and functionality that is easier to handle than ZFS - even in the case you use ZFS from CLI without Web-GUI, even if you compare the real expensive storage boxes with or without hardware raid.

If you look at the ZFS web-appliances (from FreeNAS over napp-it, NexentaStor to ZFSGuru) they are quite easy to handle and the behaviour and trouble fix procedures are easy to use and understand unless you do not have a disaster when more disks fail than you have redundancy.

But even in this case, the pool is back when enough disks come back.

I would not try with important data but ZFS should survive this with hotplug capable hardware:
Hot Unplug/ Plug any disks beyond redundancy. After you have re-inserted enough disks, the pool come back automatically.

Without hotplug capable hardware, you need a reboot. Without ZFS or with hardware raid, you have a good chance to kill the raid.

 

[2g33k4u]

i love my zfs setup if it was anything else it would have been hosed. On my Dell perc 5i I set each drive as its own vdev so the OS saw each vdev appears as a drive. then zfs assembles it all. I get the benefits of the perc card and the percs of the operating system. I trust ntfs about as much as i trust a badger in a chicken coop.