ZFSonLinux QAT acceleration in 0.7.0

vanfawx

Active Member
Jan 4, 2015
363
67
28
41
Vancouver, Canada
Posting this here in the hopes that Patrick will notice this.

With the release of ZoL 0.7.0, it is now possible to offload GZIP compression in ZFS to a QAT card. Unfortunately I do not have access to any of these, and it would be amazing to see how well it works.

Here's the closed pull request that implemented the feature:
GZIP compression offloading with QAT accelerator by wli5 · Pull Request #5836 · zfsonlinux/zfs · GitHub

And here's the changelog:
Release zfs-0.7.0 · zfsonlinux/zfs · GitHub

In addition, there's also vectorized raidz and vectorized checksums.

Anyone here have a QAT card that they can test this?
 
  • Like
Reactions: Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
11,892
4,849
113
The new drivers make QAT much easier to install @Jeggs101

Now the bigger "problem" is that Ubuntu is still on 0.6.5-9 and Proxmox is on 0.6.5.11.

Somewhat of a PITA now to setup. ZFS on Linux needs to download and install from source.

@vanfawx I am using the new Denverton C3958 board for this but have the 40Gbps QAT cards as well.
 
Last edited:
  • Like
Reactions: T_Minus and vanfawx

gigatexal

I'm here to learn
Nov 25, 2012
2,733
524
113
Portland, Oregon
alexandarnarayan.com
Yeah someone needs to kick the ZFS ubuntu maintainers in the pants and get onboard with the latest ZFS code. I almost moved my server to Fedora just for this gripe.

Also - hardware accelerated gzip ... yawn, let me know when LZ4 is hardware accelerated by these QATs.
 

sachem87184

Active Member
Feb 3, 2015
158
30
28
Sorry to bump a dead thread. But did anyone get this working with proxmox? I have a Intel 8950 QAT Card installed and setup, but the steps required to get it setup for zfs on linux are sparse.
 

sachem87184

Active Member
Feb 3, 2015
158
30
28
Looks like I have to install Debian, Compile QAT drivers, Verify QAT is working correctly, Compile ZoL libraries with --with-qat option, (looks like I need to pull the -FAST version), Verify ZFS is functional, Test QAT/ZFS combination, and then install proxmox repo and packages.

Any input here, maybe some use cases for testing?


Looks like someone made fixes for the ZoL loading before QAT when the root filesystem is running zfs:
Intel QAT support when root is on a ZFS filesystem · Issue #8323 · zfsonlinux/zfs
 
  • Like
Reactions: gigatexal

geppi

New Member
Jan 24, 2013
11
2
3
If you're on Proxmox 5.3 and have recent QAT hardware that is supported by the latest Intel QAT driver 1.7 it's sufficient to:

1. download the Intel QAT driver https://01.org/sites/default/files/downloads//qat1.7.l.4.4.0-00023.tar.gz

2. follow the instructions in the Getting Started Guide https://01.org/sites/default/files/downloads//336212qatswgettingstartedrev004.pdf

In particular I had to do:

apt-get install build-essential install libudev-dev libssl-dev pkg-config pciutils-dev pve-headers-`uname -r` libelf-dev

./configure --enable-kapi

make install

3. set the environment variable ICP_ROOT to the directory where you did the ./configure

4. apt-get install zfs-dkms

After reboot QAT accelaration for ZFS gzip compression should be available if your setup does not boot from ZFS.

If your system boots from ZFS it's currently not that easy to get the QAT accelaration because the QAT support in ZoL 0.7.12 is initialized only once when the zfs kernel modules are loaded. If you're booting from ZFS this happens from the initramfs which doesn't have the QAT drivers integrated. The patch for Intel QAT support when root is on a ZFS filesystem · Issue #8323 · zfsonlinux/zfs does check if QAT support is available also at later stages, i.e. after your system has chrooted and the QAT driver is finally available. However, this fix is currently only in the latest ZoL code and even not commited to mainline. You would need to build ZFS from the repo and install which is more trouble than the few steps above.
 
  • Like
Reactions: sachem87184

geppi

New Member
Jan 24, 2013
11
2
3
Ooops, important correction !

Step no. 4 has to be:

Add the Debian "stretch-backports" repo to your sources.list.

apt-get -t stretch-backports install zfs-dkms

This will get you ZoL 0.7.12 while the zfs-dkms from the contrib repo does only have 0.6.5.9-5 which is in conflict with 0.7.12 that you have in Proxmox 5.3.
 

sachem87184

Active Member
Feb 3, 2015
158
30
28
If you're on Proxmox 5.3 and have recent QAT hardware that is supported by the latest Intel QAT driver 1.7 it's sufficient to:

1. download the Intel QAT driver https://01.org/sites/default/files/downloads//qat1.7.l.4.4.0-00023.tar.gz

2. follow the instructions in the Getting Started Guide https://01.org/sites/default/files/downloads//336212qatswgettingstartedrev004.pdf

In particular I had to do:

apt-get install build-essential install libudev-dev libssl-dev pkg-config pciutils-dev pve-headers-`uname -r` libelf-dev

./configure --enable-kapi

make install

3. set the environment variable ICP_ROOT to the directory where you did the ./configure

4. apt-get install zfs-dkms

After reboot QAT accelaration for ZFS gzip compression should be available if your setup does not boot from ZFS.

If your system boots from ZFS it's currently not that easy to get the QAT accelaration because the QAT support in ZoL 0.7.12 is initialized only once when the zfs kernel modules are loaded. If you're booting from ZFS this happens from the initramfs which doesn't have the QAT drivers integrated. The patch for Intel QAT support when root is on a ZFS filesystem · Issue #8323 · zfsonlinux/zfs does check if QAT support is available also at later stages, i.e. after your system has chrooted and the QAT driver is finally available. However, this fix is currently only in the latest ZoL code and even not commited to mainline. You would need to build ZFS from the repo and install which is more trouble than the few steps above.
Well that stinks, I do have a mirrored rpool so I might have to wait for a commit to the repo. I've always used a rpool mirror since many of the critical files for the Proxmox VMs exist there (conf files, etc...) .

I might also look into compiling my own zfs version for giggles. Already have 2 Proxmox boxes so I might Temp move all my VMs off the QAT box and rebuild.


Edit: And I just made the connection between Geppi here and Geppi on Github! Thanks you for your commitment and work!
 

sachem87184

Active Member
Feb 3, 2015
158
30
28
So I finally got some work done on setting up my root partitions using mdadm and lvm partitions instead of zfs and took the dive in rebuilding my proxmox server.

Sadly even with all the configuration I couldn't end up getting QAT to load enough before zfs for it to be used. When I initially only had the 2 disks inserted to the root partition the zfs module didn't get loaded on boot because there were no pools for the import service to pull in. If I started the modules via modprobe I was able to see the qat kstat.

Moving forward I added several more disks the the module was then loading on boot but now I no longer have the kstat object. I even went as far as adding the intel_qat and qat_api modules into /etc/modules.d and updating the initramfs to no avail. I'm running out nof ideas on how to get it to load the modules properly so that zfs can use the QAT card.

Any help would be appreciated.


Info:

root@proxmox2:~# lsmod | grep -i zfs
zfs 3457024 11
zunicode 331776 1 zfs
zavl 16384 1 zfs
icp 258048 1 zfs
zcommon 69632 1 zfs
znvpair 77824 2 zfs,zcommon
spl 106496 4 zfs,icp,znvpair,zcommon
qat_api 536576 1 zfs

root@proxmox2:~# lsmod | grep -i qat
qat_dh895xcc 20480 0
qat_api 536576 1 zfs
intel_qat 229376 3 qat_api,usdm_drv,qat_dh895xcc
authenc 16384 1 intel_qat
uio 20480 1 intel_qat

root@proxmox2:~# modinfo zfs | grep qat
depends: spl,qat_api,znvpair,zcommon,zunicode,zavl,icp
parm: zfs_qat_disable:Disable QAT compression (int)

root@proxmox2:~# ls -al /proc/spl/kstat/zfs/
total 0
dr-xr-xr-x 3 root root 0 Feb 13 11:00 .
dr-xr-xr-x 4 root root 0 Feb 13 11:00 ..
-rw-r--r-- 1 root root 0 Feb 13 11:00 abdstats
-rw-r--r-- 1 root root 0 Feb 13 11:00 arcstats
-rw-r--r-- 1 root root 0 Feb 13 11:00 dbgmsg
-rw-r--r-- 1 root root 0 Feb 13 11:00 dbufs
-rw-r--r-- 1 root root 0 Feb 13 11:00 dmu_tx
-rw-r--r-- 1 root root 0 Feb 13 11:00 dnodestats
-rw-r--r-- 1 root root 0 Feb 13 11:00 fletcher_4_bench
-rw-r--r-- 1 root root 0 Feb 13 11:00 fm
dr-xr-xr-x 2 root root 0 Feb 13 11:00 tank
-rw-r--r-- 1 root root 0 Feb 13 11:00 vdev_cache_stats
-rw-r--r-- 1 root root 0 Feb 13 11:00 vdev_raidz_bench
-rw-r--r-- 1 root root 0 Feb 13 11:00 xuio_stats
-rw-r--r-- 1 root root 0 Feb 13 11:00 zfetchstats
-rw-r--r-- 1 root root 0 Feb 13 11:00 zil

root@proxmox2:~# service qat_service status
● qat_service.service - LSB: modprobe the QAT modules, which loads dependant modules, before calling the user space utility to pass configuration parameters
Loaded: loaded (/etc/init.d/qat_service; generated; vendor preset: enabled)
Active: active (exited) since Tue 2019-02-12 17:30:46 EST; 17h ago
Docs: man:systemd-sysv-generator(8)
Process: 27621 ExecStop=/etc/init.d/qat_service stop (code=exited, status=0/SUCCESS)
Process: 27674 ExecStart=/etc/init.d/qat_service start (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
Memory: 0B
CPU: 0
CGroup: /system.slice/qat_service.service

Feb 12 17:30:44 proxmox2 systemd[1]: Starting LSB: modprobe the QAT modules, which loads dependant modules, before calling the user space utility to pass configuration parameters...
Feb 12 17:30:45 proxmox2 qat_service[27674]: Restarting all devices.
Feb 12 17:30:45 proxmox2 qat_service[27674]: Processing /etc/dh895xcc_dev0.conf
Feb 12 17:30:46 proxmox2 qat_service[27674]: Checking status of all devices.
Feb 12 17:30:46 proxmox2 qat_service[27674]: There is 1 QAT acceleration device(s) in the system:
Feb 12 17:30:46 proxmox2 qat_service[27674]: qat_dev0 - type: dh895xcc, inst_id: 0, node_id: 0, bsf: 0000:06:00.0, #accel: 6 #engines: 12 state: up
Feb 12 17:30:46 proxmox2 systemd[1]: Started LSB: modprobe the QAT modules, which loads dependant modules, before calling the user space utility to pass configuration parameters.


root@proxmox2:/proc# cat /sys/module/zfs/parameters/zfs_qat_disable
0
 

geppi

New Member
Jan 24, 2013
11
2
3
What do you get if you run:

lsinitramfs -l /boot/initrd.img-`uname -r` |grep zfs.ko

?
 

sachem87184

Active Member
Feb 3, 2015
158
30
28
lsinitramfs -l /boot/initrd.img-4.15.18-11-pve | grep -i zfs
-rw-r--r-- 1 root root 28 Feb 12 10:40 etc/modprobe.d/zfs.conf


No ko loaded


root@proxmox2:/boot# cat /etc/modprobe.d/zfs.conf
blacklist spl
blacklist zfs