How to speed up resilver? (ZFS + Ubuntu bionic)

Discussion in 'Linux Admins, Storage and Virtualization' started by mrjayviper, Mar 23, 2019.

  1. mrjayviper

    mrjayviper New Member

    Joined:
    Jul 28, 2017
    Messages:
    21
    Likes Received:
    2
    My setup:

    • 8x 2TB RAIDZ -was built with ashift=12 (confirmed using zdb)
    • using built-in SATA ports (AMD chipset with 8 ports)
    I replaced 1 drive with a 3TB one and the resilver rate is around 5.x MBps. According to the status, it will take around 240 hours to complete!

    Am I doing something wrong?

    I can rebuilt/destroy the array as I've copied array contents to other disks.

    Thanks
     
    #1
  2. zxv

    zxv The more I C, the less I see.

    Joined:
    Sep 10, 2017
    Messages:
    152
    Likes Received:
    46
    If it's a single vdev (8 disks as a single raidz stripe), then yes, that will be slower than say, 4 mirrored pairs. But even so, 5MB/s seems low.

    Can you paste the output of "zpool status -v" and "zfs get all <poolname>" and we can try to see what's going on.
     
    #2
  3. fjes82

    fjes82 New Member

    Joined:
    Jan 4, 2020
    Messages:
    1
    Likes Received:
    2
    For the record I had the same problem and spend 2 nights very worried.

    I tried everything: replace, detach/attach, reboots, adjust parameters (resilver_delay, resilver_min_time_ms, top_maxinflight) and tweeking zfs_vdev_async_write_min_active (which was already set to good values in my ubuntu 18.04 repo defaults) without any improvement.

    In the end, I discovered that my replaced disk was too hot (it was hitting 72ÂșC, the limit for my ST8000VN disk), as reported by smartctl tests. I couldnt even touch it, but could very well fry some eggs in it.

    I installed him very close to other disks and ventilation was insufficient at that moment.

    After I solved these issues, my 2,67TB resilver came from 562h to 13h.

    Strangely, "iostat -x" reported near 99% usage with very little data transfer in the first scenario, and approximately 20% usage in the latter.

    Hope it helps any poor soul wasting his weekend like I did.
     
    #3
    vanfawx and Albert Yang like this.
  4. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    321
    Likes Received:
    57
    As well, ZFS 0.8 has introduced a more sequential scrub/resilver logic that should improve things as well. On one of my systems, resilver went from 8 hours to 3.
     
    #4
  5. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,290
    Likes Received:
    758
    There are tuning options for resilver time but with a very little improvement. Resilver must read all metadata and data what means that resilver time is traditionally mostly related to the pool performance for small io.

    To improve this, Oracle introduced sequential resilvering in 2015 what means that first metadate is read to sort datareads to make them more sequential than random, Sequential Resilvering

    This improvement is now available in Open-ZFS (current Illumos, Free-BSD and ZoL) and called sorted resilvering. Only this can reduce resilver/scrub time dramatically.
     
    #5
  6. vangoose

    vangoose Member

    Joined:
    May 21, 2019
    Messages:
    69
    Likes Received:
    25
    I just did a scrub on 24 * 3 TB disk zpool, 3*raidz2 vdev(8 disk), 28TB data, Resilver rate is 1.5GB/s
    CentOS 8, zfs 0.82
     
    #6
Similar Threads: speed resilver
Forum Title Date
Linux Admins, Storage and Virtualization Slow SMB Read Speed, Fast Write | ZFS on Linux | Ubuntu 18.10 Nov 4, 2018
Linux Admins, Storage and Virtualization Reduce SAS link speed to 6G ? Jun 8, 2018
Linux Admins, Storage and Virtualization Mild brain picking (PXE boot multi OS from server, ethernet speed for virtualization, & more) Feb 1, 2017
Linux Admins, Storage and Virtualization Need help with slow samba file copy speeds Nov 7, 2016
Linux Admins, Storage and Virtualization Help! ZFS resilver ended with "insufficient replicas" Mar 17, 2017

Share This Page