ZFSonLinux QAT acceleration in 0.7.0

Discussion in 'Linux Admins, Storage and Virtualization' started by vanfawx, Sep 15, 2017.

  1. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    304
    Likes Received:
    52
    Posting this here in the hopes that Patrick will notice this.

    With the release of ZoL 0.7.0, it is now possible to offload GZIP compression in ZFS to a QAT card. Unfortunately I do not have access to any of these, and it would be amazing to see how well it works.

    Here's the closed pull request that implemented the feature:
    GZIP compression offloading with QAT accelerator by wli5 · Pull Request #5836 · zfsonlinux/zfs · GitHub

    And here's the changelog:
    Release zfs-0.7.0 · zfsonlinux/zfs · GitHub

    In addition, there's also vectorized raidz and vectorized checksums.

    Anyone here have a QAT card that they can test this?
     
    #1
    Patrick likes this.
  2. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,576
    Likes Received:
    4,522
    Just got a QAT C3958 Gigabyte board. Will try when I can.
     
    #2
    niekbergboer likes this.
  3. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    304
    Likes Received:
    52
    Woot! Thank you very much @Patrick!
     
    #3
  4. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,466
    Likes Received:
    216
    Installing is still a bit painful on Ubuntu but it is getting there. Almost had QAT working this morning, just did a fresh install.
     
    #4
  5. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    304
    Likes Received:
    52
    Do you have a QAT card, or is it embedded? I was looking to buy the AIC, but it's much more expensive than I was expecting.
     
    #5
  6. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    304
    Likes Received:
    52
    Also sweet! Thank you! :D
     
    #6
  7. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,466
    Likes Received:
    216
    I have the QAT card from Intel. I now see @Patrick used CentOS
     
    #7
    vanfawx and Patrick like this.
  8. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,576
    Likes Received:
    4,522
    The new drivers make QAT much easier to install @Jeggs101

    Now the bigger "problem" is that Ubuntu is still on 0.6.5-9 and Proxmox is on 0.6.5.11.

    Somewhat of a PITA now to setup. ZFS on Linux needs to download and install from source.

    @vanfawx I am using the new Denverton C3958 board for this but have the 40Gbps QAT cards as well.
     
    #8
    Last edited: Sep 16, 2017
    T_Minus and vanfawx like this.
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,576
    Likes Received:
    4,522
    OK - ZFS 0.7.2 + QAT is now on a C3958 system. Finally!
     
    #9
    nitrobass24 likes this.
  10. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    304
    Likes Received:
    52
    I'm excited! :D
     
    #10
  11. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,697
    Likes Received:
    500
    Yeah someone needs to kick the ZFS ubuntu maintainers in the pants and get onboard with the latest ZFS code. I almost moved my server to Fedora just for this gripe.

    Also - hardware accelerated gzip ... yawn, let me know when LZ4 is hardware accelerated by these QATs.
     
    #11
  12. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    304
    Likes Received:
    52
    @gigatexal gzip has a superior compression ratio to LZ4, and offloaded, has a performance advantage as well. Don't get me wrong, I love LZ4, but if I had the choice, I'd pick gzip-9 over LZ4.
     
    #12
    gigatexal likes this.
  13. sachem87184

    sachem87184 Active Member

    Joined:
    Feb 3, 2015
    Messages:
    157
    Likes Received:
    29
    Sorry to bump a dead thread. But did anyone get this working with proxmox? I have a Intel 8950 QAT Card installed and setup, but the steps required to get it setup for zfs on linux are sparse.
     
    #13
  14. sachem87184

    sachem87184 Active Member

    Joined:
    Feb 3, 2015
    Messages:
    157
    Likes Received:
    29
    Looks like I have to install Debian, Compile QAT drivers, Verify QAT is working correctly, Compile ZoL libraries with --with-qat option, (looks like I need to pull the -FAST version), Verify ZFS is functional, Test QAT/ZFS combination, and then install proxmox repo and packages.

    Any input here, maybe some use cases for testing?


    Looks like someone made fixes for the ZoL loading before QAT when the root filesystem is running zfs:
    Intel QAT support when root is on a ZFS filesystem · Issue #8323 · zfsonlinux/zfs
     
    #14
    gigatexal likes this.
  15. geppi

    geppi New Member

    Joined:
    Jan 24, 2013
    Messages:
    9
    Likes Received:
    2
    If you're on Proxmox 5.3 and have recent QAT hardware that is supported by the latest Intel QAT driver 1.7 it's sufficient to:

    1. download the Intel QAT driver https://01.org/sites/default/files/downloads//qat1.7.l.4.4.0-00023.tar.gz

    2. follow the instructions in the Getting Started Guide https://01.org/sites/default/files/downloads//336212qatswgettingstartedrev004.pdf

    In particular I had to do:

    apt-get install build-essential install libudev-dev libssl-dev pkg-config pciutils-dev pve-headers-`uname -r` libelf-dev

    ./configure --enable-kapi

    make install

    3. set the environment variable ICP_ROOT to the directory where you did the ./configure

    4. apt-get install zfs-dkms

    After reboot QAT accelaration for ZFS gzip compression should be available if your setup does not boot from ZFS.

    If your system boots from ZFS it's currently not that easy to get the QAT accelaration because the QAT support in ZoL 0.7.12 is initialized only once when the zfs kernel modules are loaded. If you're booting from ZFS this happens from the initramfs which doesn't have the QAT drivers integrated. The patch for Intel QAT support when root is on a ZFS filesystem · Issue #8323 · zfsonlinux/zfs does check if QAT support is available also at later stages, i.e. after your system has chrooted and the QAT driver is finally available. However, this fix is currently only in the latest ZoL code and even not commited to mainline. You would need to build ZFS from the repo and install which is more trouble than the few steps above.
     
    #15
    sachem87184 likes this.
  16. geppi

    geppi New Member

    Joined:
    Jan 24, 2013
    Messages:
    9
    Likes Received:
    2
    Ooops, important correction !

    Step no. 4 has to be:

    Add the Debian "stretch-backports" repo to your sources.list.

    apt-get -t stretch-backports install zfs-dkms

    This will get you ZoL 0.7.12 while the zfs-dkms from the contrib repo does only have 0.6.5.9-5 which is in conflict with 0.7.12 that you have in Proxmox 5.3.
     
    #16
  17. sachem87184

    sachem87184 Active Member

    Joined:
    Feb 3, 2015
    Messages:
    157
    Likes Received:
    29
    Well that stinks, I do have a mirrored rpool so I might have to wait for a commit to the repo. I've always used a rpool mirror since many of the critical files for the Proxmox VMs exist there (conf files, etc...) .

    I might also look into compiling my own zfs version for giggles. Already have 2 Proxmox boxes so I might Temp move all my VMs off the QAT box and rebuild.


    Edit: And I just made the connection between Geppi here and Geppi on Github! Thanks you for your commitment and work!
     
    #17
  18. sachem87184

    sachem87184 Active Member

    Joined:
    Feb 3, 2015
    Messages:
    157
    Likes Received:
    29
    So I finally got some work done on setting up my root partitions using mdadm and lvm partitions instead of zfs and took the dive in rebuilding my proxmox server.

    Sadly even with all the configuration I couldn't end up getting QAT to load enough before zfs for it to be used. When I initially only had the 2 disks inserted to the root partition the zfs module didn't get loaded on boot because there were no pools for the import service to pull in. If I started the modules via modprobe I was able to see the qat kstat.

    Moving forward I added several more disks the the module was then loading on boot but now I no longer have the kstat object. I even went as far as adding the intel_qat and qat_api modules into /etc/modules.d and updating the initramfs to no avail. I'm running out nof ideas on how to get it to load the modules properly so that zfs can use the QAT card.

    Any help would be appreciated.


    Info:

    root@proxmox2:~# lsmod | grep -i zfs
    zfs 3457024 11
    zunicode 331776 1 zfs
    zavl 16384 1 zfs
    icp 258048 1 zfs
    zcommon 69632 1 zfs
    znvpair 77824 2 zfs,zcommon
    spl 106496 4 zfs,icp,znvpair,zcommon
    qat_api 536576 1 zfs

    root@proxmox2:~# lsmod | grep -i qat
    qat_dh895xcc 20480 0
    qat_api 536576 1 zfs
    intel_qat 229376 3 qat_api,usdm_drv,qat_dh895xcc
    authenc 16384 1 intel_qat
    uio 20480 1 intel_qat

    root@proxmox2:~# modinfo zfs | grep qat
    depends: spl,qat_api,znvpair,zcommon,zunicode,zavl,icp
    parm: zfs_qat_disable:Disable QAT compression (int)

    root@proxmox2:~# ls -al /proc/spl/kstat/zfs/
    total 0
    dr-xr-xr-x 3 root root 0 Feb 13 11:00 .
    dr-xr-xr-x 4 root root 0 Feb 13 11:00 ..
    -rw-r--r-- 1 root root 0 Feb 13 11:00 abdstats
    -rw-r--r-- 1 root root 0 Feb 13 11:00 arcstats
    -rw-r--r-- 1 root root 0 Feb 13 11:00 dbgmsg
    -rw-r--r-- 1 root root 0 Feb 13 11:00 dbufs
    -rw-r--r-- 1 root root 0 Feb 13 11:00 dmu_tx
    -rw-r--r-- 1 root root 0 Feb 13 11:00 dnodestats
    -rw-r--r-- 1 root root 0 Feb 13 11:00 fletcher_4_bench
    -rw-r--r-- 1 root root 0 Feb 13 11:00 fm
    dr-xr-xr-x 2 root root 0 Feb 13 11:00 tank
    -rw-r--r-- 1 root root 0 Feb 13 11:00 vdev_cache_stats
    -rw-r--r-- 1 root root 0 Feb 13 11:00 vdev_raidz_bench
    -rw-r--r-- 1 root root 0 Feb 13 11:00 xuio_stats
    -rw-r--r-- 1 root root 0 Feb 13 11:00 zfetchstats
    -rw-r--r-- 1 root root 0 Feb 13 11:00 zil

    root@proxmox2:~# service qat_service status
    ● qat_service.service - LSB: modprobe the QAT modules, which loads dependant modules, before calling the user space utility to pass configuration parameters
    Loaded: loaded (/etc/init.d/qat_service; generated; vendor preset: enabled)
    Active: active (exited) since Tue 2019-02-12 17:30:46 EST; 17h ago
    Docs: man:systemd-sysv-generator(8)
    Process: 27621 ExecStop=/etc/init.d/qat_service stop (code=exited, status=0/SUCCESS)
    Process: 27674 ExecStart=/etc/init.d/qat_service start (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
    Memory: 0B
    CPU: 0
    CGroup: /system.slice/qat_service.service

    Feb 12 17:30:44 proxmox2 systemd[1]: Starting LSB: modprobe the QAT modules, which loads dependant modules, before calling the user space utility to pass configuration parameters...
    Feb 12 17:30:45 proxmox2 qat_service[27674]: Restarting all devices.
    Feb 12 17:30:45 proxmox2 qat_service[27674]: Processing /etc/dh895xcc_dev0.conf
    Feb 12 17:30:46 proxmox2 qat_service[27674]: Checking status of all devices.
    Feb 12 17:30:46 proxmox2 qat_service[27674]: There is 1 QAT acceleration device(s) in the system:
    Feb 12 17:30:46 proxmox2 qat_service[27674]: qat_dev0 - type: dh895xcc, inst_id: 0, node_id: 0, bsf: 0000:06:00.0, #accel: 6 #engines: 12 state: up
    Feb 12 17:30:46 proxmox2 systemd[1]: Started LSB: modprobe the QAT modules, which loads dependant modules, before calling the user space utility to pass configuration parameters.


    root@proxmox2:/proc# cat /sys/module/zfs/parameters/zfs_qat_disable
    0
     
    #18
  19. geppi

    geppi New Member

    Joined:
    Jan 24, 2013
    Messages:
    9
    Likes Received:
    2
    What do you get if you run:

    lsinitramfs -l /boot/initrd.img-`uname -r` |grep zfs.ko

    ?
     
    #19
  20. sachem87184

    sachem87184 Active Member

    Joined:
    Feb 3, 2015
    Messages:
    157
    Likes Received:
    29
    lsinitramfs -l /boot/initrd.img-4.15.18-11-pve | grep -i zfs
    -rw-r--r-- 1 root root 28 Feb 12 10:40 etc/modprobe.d/zfs.conf


    No ko loaded


    root@proxmox2:/boot# cat /etc/modprobe.d/zfs.conf
    blacklist spl
    blacklist zfs
     
    #20
Similar Threads: ZFSonLinux acceleration
Forum Title Date
Linux Admins, Storage and Virtualization ZFSonLinux (aka ZoL) now at version 0.7 - big update imo Aug 3, 2017

Share This Page