How important is QAT for distributed storage?

wuubb

New Member
May 7, 2022
5
3
3
Specifically, I'm looking at Ceph (via Rook) and TrueNAS Scale (so OpenZFS) between CPUs like Xeon D 1500/AMD Epyc 3000 vs Atom C3000/Xeon D 1700.

I can't seem to find anything about using QAT with Ceph/ZFS besides the fact that it's available to use. There is an older thread here about QAT on OpenZFS but it seems to be all about install headaches on earlier kernels and no actual evaluation.

Does anyone have any insight on whether QAT is worth it for distributed storage setups? I would be encrypting the ZFS datasets and Ceph OSDs (which it does not do by default).
 
Last edited:

Stephan

Well-Known Member
Apr 21, 2017
559
356
63
Germany
In my unscientific opinion, not worth it. Also smells like vendor lock-in. Want to switch from Intel to ARM64, but the algorithm is a bit proprietary. What now?

For encryption there is AES acceleration in the CPU for a bunch of years now, which is really fast. Makes QAT worthless. Want something else like ChaCha20-Poly1305, or something post-quantum-secure, you would pick a vectorized (AVX2...) library anyhow, because QAT is stuck with the old stuff.

For compression, you can choose LZ4 which for at least decompression is basically only 2.75 times slower than memcpy. Copy anything around in a buffer to feed QAT and you are already losing. Or if you want better compression choose zstd at 2700 MByte/s decompression speed. At queue depth = 1 this means a single modern x86 core can keep up with your single SSD and decompress everything with mostly no penalty on the fly.

Bottom line, modern CPUs from Skylake onwards are powerful enough to skip QAT. There used to be a time when this was useful. Like when i486DX2-66 where the fastest PC CPUs around. These profited from offloading bigtime.