potential data loss bug in ZFS on Linux 0.7.7

Discussion in 'Linux Admins, Storage and Virtualization' started by BLinux, Apr 8, 2018.

  1. BLinux

    BLinux Well-Known Member

    Joined:
    Jul 7, 2016
    Messages:
    1,249
    Likes Received:
    256
    for anyone using latest ZFS on Linux, I think you'll want to know about this bug:

    Unlistable and disappearing files · Issue #7401 · zfsonlinux/zfs

    highlights: although this bug is easily reproduced on CentOS/RHEL distros, it also affects Ubuntu and others. people have found ways to trigger this bug in non-CentOS/RHEL OSes now...

    on CentOS 7, to revert back to 0.7.6, i've confirmed this command works:

    $ yum downgrade kmod-spl kmod-zfs libnvpair1 libuutil1 libzfs2 libzpool2 spl zfs

    this was a clean install with 0.7.7 installed, not upgraded from previous, and I confirmed I can downgrade with the above command.
     
    #1
    Last edited: Apr 9, 2018
    T_Minus likes this.
  2. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,334
    Likes Received:
    382
    Seems only to affect RHEL and then those from the repos as building it from source on Ubuntu doesn’t show this issue.

    Also yum is transactional I had no idea you could do this:

    For reverts, I usually do:

    $ yum history (identify transaction that installed 0.7.7 over 0.7.6; yum history info XXX can be used to confirm)
    $ yum history undo XXX (where XXX is the transaction number identified in the previous step)
    Note that with dkms installs, after reverts, I usually find I need to:

    $ dkms remove zfs/0.7.6 -k `uname -r`
    $ dkms remove spl/0.7.6 -k `uname -r`
    $ dkms install spl/0.7.6 -k `uname -r` --force
    $ dkms install zfs/0.7.6 -k `uname -r` --force
    To make sure all modules are actually happy and loadable on reboot.


    Sent from my iPhone using Tapatalk
     
    #2
  3. BLinux

    BLinux Well-Known Member

    Joined:
    Jul 7, 2016
    Messages:
    1,249
    Likes Received:
    256
    yeah, downgrade is easy. and even if you installed straight to 0.7.7 (without upgrading from previous version), you can downgrade with:

    yum downgrade kmod-spl kmod-zfs libnvpair1 libuutil1 libzfs2 libzpool2 spl zfs

    and it will take you to 1 version eariler as long as the RPMs are available in the repo.
     
    #3
    T_Minus and Evan like this.
  4. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,334
    Likes Received:
    382
    That’s really hella cool. Is there a Ubuntu/Debian feature that does the same?


    Sent from my iPhone using Tapatalk
     
    #4
    cactus likes this.
  5. BLinux

    BLinux Well-Known Member

    Joined:
    Jul 7, 2016
    Messages:
    1,249
    Likes Received:
    256
    sounds like they are about to pull the 0.7.7 RPMs and will release a 0.7.8. current recommended course of action is to downgrade to 0.7.6 until this is fixed and 0.7.8 is released.
     
    #5
  6. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,782
    Likes Received:
    3,761
  7. BLinux

    BLinux Well-Known Member

    Joined:
    Jul 7, 2016
    Messages:
    1,249
    Likes Received:
    256
    latest update: they've reproduced this bug on other non-RHEL/CentOS distros... so if using ZoL on Ubuntu or others, you need to take precautions too...
     
    #7
  8. BLinux

    BLinux Well-Known Member

    Joined:
    Jul 7, 2016
    Messages:
    1,249
    Likes Received:
    256
    @Patrick @cliffr : you might want to update the article as other OSes are also affected. the statement "It seems that users of Ubuntu among others may be safe from this one" is no longer true...
     
    #8
    T_Minus likes this.
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    10,782
    Likes Received:
    3,761
    @BLinux thanks for the tip. Updated.
     
    #9
  10. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,639
    Likes Received:
    1,032
    Note that you don't get ZFS 0.7.7 as part of Ubuntu. With the last official release (17.10) they only packaged ZFS 0.6.x (I think 0.6.5) and even though it's not released officially for a couple more weeks the 18.04/Bionic packages only include ZFS 0.7.5. There are unofficial repositories (PPAs) and the self-build option to get ZFS 0.7.7. But an Ubuntu user using the "official" packages from Canonical isn't at risk because they never integrated the buggy release.

    Lots of Proxmox users here too - I believe (need to check, can't VPN my lab right now) Proxmox 5.1 includes ZFS 0.7.2.
     
    #10
    gigatexal, T_Minus and Patrick like this.
  11. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,334
    Likes Received:
    382
    I think this is a level one crisis for the project. It has to get fixed fast so as to not damage the good name of ZFS


    Sent from my iPhone using Tapatalk
     
    #11
  12. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,177
    Likes Received:
    265
    #12
    niekbergboer and gigatexal like this.
  13. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,639
    Likes Received:
    1,032
    #13
    niekbergboer and Patrick like this.
  14. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    823
    Likes Received:
    82
    Guess running VM's never went into this, as it obviously happened with large numbers of small files.
     
    #14
Similar Threads: potential data
Forum Title Date
Linux Admins, Storage and Virtualization Getting SMART data from ZFS disks in Proxmox Mar 10, 2018
Linux Admins, Storage and Virtualization Shared Multi-Host Storage for Docker and Data Volumes Jul 13, 2017
Linux Admins, Storage and Virtualization Colo, Datatcenter: ease of managing life Apr 24, 2017
Linux Admins, Storage and Virtualization unlimited cloud backup for large data sets for linux? Aug 8, 2016
Linux Admins, Storage and Virtualization Meta data corruption on SuperMicro H8DG6 Apr 1, 2016

Share This Page