raid-Z1 (3 drive) to two-mirrored vdevs (4 drives) - problem!

[mixer]

I am in the process of 'physical-izing' my OI/napp-it ZFS base and host my VMs in KVM on OI instead of ESXi.

Part of the update is changing my 3-drive raid-z of 3TB WD REDs over to 4 drives, a pool made of two vdevs made up of two-drive mirror each (how do you say that more concisely?).

Initial testing with Bonnie ++ indicates a problem. I have 32 GB of RAM and the Bonnie run took about 4 hours with this result:

NAME SIZE  Bonnie Date(y.m.d) File Seq-Wr-Chr %CPU Seq-Write %CPU Seq-Rewr %CPU Seq-Rd-Chr %CPU Seq-Read %CPU Rnd Seeks %CPU Files Seq-Create Rnd-Create
reds 5.44T start  2013.01.28  64G  56 MB/s    38   22 MB/s    2   13 MB/s   2   21 MB/s    17   33 MB/s  1    1291.2/s   3   16    22692/s    +++++/s

Obviously much much much slower than before. I can think of a few bits of trouble-shooting to do, but is this anything you've seen before?

Thanks. I'll update, probably with a solution soon

-- update. hard errors. a cable maybe, I'm hoping. That was not the new drive!:

c4t2d0  	 1  	 via dd  	 ok  	 3001 GB  	 3 TB  	 2.7 TiB  	  S:0 H:0 T:0        ATA  	 WDC WD30EFRX-68A  	 WDWMC1Txxxxxxx 
c4t3d0  	 1  	 via dd  	 ok  	 3001 GB  	 3 TB  	 2.7 TiB  	  S:0 H:0 T:0        ATA  	 WDC WD30EFRX-68A  	 WDWMC1Txxxxxxx 
c4t4d0  	 1  	 via dd  	 ok  	 3001 GB  	 3 TB  	 2.7 TiB  	  S:0 H:587 T:0      ATA  	 WDC WD30EFRX-68A  	 WDWMC1Txxxxxxx 
c4t5d0  	 1  	 via dd  	 ok  	 3001 GB  	 3 TB  	 2.7 TiB  	  S:0 H:0 T:0        ATA  	 WDC WD30EFRX-68A  	 WDWMC1Txxxxxxx

 

[Patrick]

Ah ha! Good troubleshooting. Keep us posted. Probably best to resolve that then see.

 

[mixer]

It's not just the cable. SMART short test fails on that drive with "read error" and SMART attribute Raw Read Error Rate is 3283 and Current Pending Sector is 51. RMA time I guess.

 

[mixer]

Getting there... here's with three disks, one mirror vdev and one single vdev (waiting for its mirror mate to arrive from WD advance RMA):

NAME SIZE  Bonnie Date(y.m.d) File Seq-Wr-Chr %CPU Seq-Write %CPU Seq-Rewr %CPU Seq-Rd-Chr %CPU Seq-Read %CPU Rnd Seeks %CPU Files Seq-Create Rnd-Create

reds 5.44T start  2013.01.28  64G  56 MB/s    38   22 MB/s    2   13 MB/s   2   21 MB/s    17   33 MB/s  1    1291.2/s   3   16    22692/s    +++++/s
reds 5.44T start  2013.01.30  64G  147 MB/s   99   224 MB/s   20  111 MB/s  19  128 MB/s   92  252 MB/s 13    1004.0/s   2   16    22558/s    +++++/s

and compared to the RAID-Z

reds 8.12T start  2012.10.11  24G  139 MB/s   99   230 MB/s   17  115 MB/s  12  132 MB/s   99  281 MB/s 9    661.7/s     1   16   +++++/s 	 +++++/s

Performance should of course improve when the 4th disk is back.. I'll update! The main reason for this post was to compare RaidZ vs two two-way mirrors.