ZFS Speeds - Silver and scrub

[wookienz]

Hi all,

I built a ZFS server a couple of years ago, basic details are:

Norco 4224
3 x IBM 1015 HBA mode connected to..
24 x 2TB HDD's (WD Greens)
Xeon e52620 CPU

Vmware hypervisor
openindiana with nappit

I have 3 x 6 HDD z2 vdevs connected to one pool

Silvering and Scrubbing are crazy slow, somewhere in the 1-2MB/s region with the pool being used by other VM's. With absolutely nothing connected i can get 30MB/s during a resilver.

All drives appear ok on smartctl.

It doesnt appear right. I would appreciate any help to troubleshoot and guidance of where to start.


Cheers.

 

[gea]

A scrub is a process that runs at a low priority.
Low values under load are normal

Without load, scrub performance depend on IO/disk performance (WD Greens are slow)
and on pool fillrate and especially on amount of small files .
NexentaStor Project - URGENT help need - resilver killed my san - NexentaStor Project

What you can do
- check iostat if all disks offer similar read/write values or if some are worse or have high transfer/hard errors
- check menu system-log and system-faults

- increase scub priority
ZFS: Performance Tuning for Scrubs and Resilvers | Jason Matthews' Technical Operations

 

[wookienz]

Thanks for the reply.

Woudnt you know it, after i shut down the VM's that use NFS shares off the ZFS storage and rebooted them, the resilver rate jumper to 150MB/s. Makes no sense as the VM's are using the nfs shares constantly so it should slow down the resilver, but instead it speed it up.

I have seen his blog. I used the resilver and scrub command, didint seem to change the scrub and i hadnt checked the resilver before i changed it. Presently resilver looks good, i guess ill have to wait and then scrub the data set and see what i get after it magically appears to be running faster.

Ill dump so iostat and see what you think it is all greek to me.

 

[wookienz]

System Logs:

Jun 29 19:24:58 openindiana cmlb: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g50014ee6007b7c8b (sd2):
Jun 29 19:24:58 openindiana primary label corrupt; using backup
Jun 29 19:24:58 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:24:58 openindiana Aborted_command!
Jun 29 19:24:59 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:24:59 openindiana Aborted_command!
Jun 29 19:24:59 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:24:59 openindiana Aborted_command!
Jun 29 19:24:59 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:24:59 openindiana Aborted_command!
Jun 29 19:24:59 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:24:59 openindiana Aborted_command!
Jun 29 19:25:00 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:00 openindiana Aborted_command!
Jun 29 19:25:00 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:00 openindiana Aborted_command!
Jun 29 19:25:00 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:00 openindiana Aborted_command!
Jun 29 19:25:00 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:00 openindiana Aborted_command!
Jun 29 19:25:01 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:01 openindiana Aborted_command!
Jun 29 19:25:01 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:01 openindiana Aborted_command!
Jun 29 19:25:01 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:01 openindiana Aborted_command!
Jun 29 19:25:01 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:01 openindiana Aborted_command!
Jun 29 19:25:01 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:01 openindiana Aborted_command!
Jun 29 19:25:01 openindiana scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:01 openindiana mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110d01
Jun 29 19:25:01 openindiana scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:01 openindiana mptsas_handle_event: IOCStatus=0x8000, IOCLogInfo=0x31110d01
Jun 29 19:25:02 openindiana scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:02 openindiana Log info 0x31110d01 received for target 14.
Jun 29 19:25:02 openindiana scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc
Jun 29 19:25:02 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:02 openindiana Aborted_command!
Jun 29 19:25:02 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:02 openindiana Aborted_command!
Jun 29 19:25:03 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:03 openindiana Aborted_command!
Jun 29 19:25:03 openindiana scsi: [ID 107833 kern.notice] /pci@0,0/pci15ad,7a0@15/pci1000,3020@0 (mpt_sas0):
Jun 29 19:25:03 openindiana Aborted_command!
Jun 29 19:25:48 openindiana mDNSResponder: [ID 702911 daemon.error] ERROR: getOptRdata - unknown opt 4
Jun 29 19:26:40 openindiana last message repeated 6 times
Jun 29 20:34:56 openindiana mDNSResponder: [ID 702911 daemon.error] ERROR: getOptRdata - unknown opt 4
Jun 29 20:35:03 openindiana last message repeated 3 times

 

[wookienz]

scott@openindiana:~$ iostat -enx
extended device statistics ---- errors ---
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 fd0
0.5 1.7 5.9 9.2 0.0 0.0 0.0 0.3 0 0 0 0 0 0 c4t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 5 0 5 c3t0d0
3.7 14.5 152.1 82.1 0.0 0.2 0.0 9.0 0 5 4 21 21 46 c5t50014EE6007B7C8Bd0
3.3 12.1 160.8 65.9 0.0 0.2 0.0 14.0 0 5 4 2 11 17 c5t50014EE25C8CB00Fd0
4.4 14.8 146.1 82.3 0.0 0.2 0.0 8.3 0 5 4 2 6 12 c5t50014EE2B1E267E0d0
3.6 11.9 169.3 65.7 0.0 0.2 0.0 12.7 0 5 4 6 15 25 c5t50014EE25C7570A9d0
3.9 14.5 155.2 82.0 0.0 0.2 0.0 8.5 0 4 4 0 0 4 c5t50014EE15A152551d0
3.5 11.9 167.7 65.7 0.0 0.2 0.0 13.5 0 5 4 0 0 4 c5t50014EE1AF64263Bd0
4.1 14.5 161.3 82.0 0.0 0.2 0.0 8.7 0 4 4 2 5 11 c5t50014EE207378C1Bd0
3.4 12.1 161.7 65.9 0.0 0.2 0.0 12.7 0 5 4 0 0 4 c5t50014EE2071A2ED9d0
3.6 14.6 149.8 82.1 0.0 0.2 0.0 8.3 0 4 4 0 0 4 c5t50014EE1AF64271Dd0
4.1 11.8 157.0 65.6 0.0 0.2 0.0 12.4 0 5 4 0 0 4 c5t50014EE207202746d0
4.2 14.9 149.8 82.4 0.0 0.2 0.0 8.4 0 5 4 0 0 4 c5t50014EE2B1E2690Dd0
4.1 11.8 156.4 65.6 0.0 0.2 0.0 12.2 0 5 4 0 0 4 c5t50014EE2071A3945d0
0.0 0.0 2.4 0.0 0.0 0.0 0.0 2.6 0 0 0 0 0 0 c5t50014EE2067D0FE4d0
0.0 0.0 2.4 0.0 0.0 0.0 0.0 2.5 0 0 0 0 0 0 c5t50014EE6AB2605ECd0
0.0 0.3 2.4 10.2 0.0 0.0 0.0 18.7 0 0 0 0 0 0 c5t50014EE002762C0Ad0
0.0 0.0 2.4 0.0 0.0 0.0 0.7 2.6 0 0 1 1891 225 2117 c5t50014EE655D0EFE8d0
0.0 0.0 2.4 0.0 0.0 0.0 0.0 2.4 0 0 0 0 0 0 c5t50014EE0AD20FFF6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0 0 0 0 0 0 c5t50014EE207378533d0
5.0 19.7 248.5 294.7 0.0 0.2 0.0 9.2 0 6 4 0 0 4 c5t5000C500386F3796d0
4.9 19.9 241.2 295.1 0.0 0.2 0.0 8.8 0 6 4 0 0 4 c5t5000C500380ED667d0
4.2 19.8 198.8 294.9 0.0 0.3 0.2 12.0 0 7 5 36484 195126 231615 c5t5000C50037FBB336d0
5.1 19.8 250.3 294.8 0.0 0.2 0.0 9.2 0 6 4 0 0 4 c5t5000C500380F9535d0
4.9 19.8 238.1 294.9 0.0 0.2 0.0 8.7 0 6 4 0 0 4 c5t5000C500380CBDA6d0
4.4 19.8 208.0 294.9 0.0 0.2 0.0 8.5 0 6 4 3 10 17 c5t5000C5003806FE93d0


note: c5t5000C50037FBB336d0 is being replaced and resilvered presently. c5t50014EE655D0EFE8d0 may have to be done as well.

Any better iostat commands to run?