OmniOS ZFS encryption on Intel Xeon silver 4110 vs AMD Epyc 7302 on a SM H12SSL-C

gea

Well-Known Member
Dec 31, 2010
2,870
1,013
113
DE
After weeks of waiting I got a new SuperMicro BTO system with the H12SSL-C mainboard and a 16/32 Core 7302 and 128GB RAM, H12SSL-C | Motherboards | Super Micro Computer, Inc.

The same board is also available with 10G nics (but not yet supported on my target OS OmniOS)

I made some tests with a disk pool, an NVMe pool and an Optane pool with and without encryption as this is what requires performance. While sync write seems to have a performance problem with encryption, the other results are promising. As you can load such a system with 24 NVMe system performance may need a jump.

I have also tried a virtualized OmniOS on ESXi 7.0U1 but found NVMe pass-through problems that need some more work. 12G SAS in pass-through mode ex with WD SS530 SAS SSDs are troublefree and nearly as fast as NVMe.

First impression: Up to twice as fast with the Epyc system and same pool compared to a top system from 2019 https://napp-it.org/doc/downloads/epyc_performance.pdf
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
2,870
1,013
113
DE
On non sync writes you can encrypt quite large datablocks in recsize ex 128k what works fast. I asume that the encryption of very small datablocks (logging of every committed write) down to 4k physical disk blocksize is not efficient.
 

Lahey

New Member
Nov 28, 2020
2
0
1
I just ran some benchmarks on a OmniOS/napp-it host the other day and was quite surprised at how CPU expensive zfs + encryption actually is compared to a similar linux based setup (mdadm, LUKS with aes-256-xts, ext4, NFS share) on the same hardware. After testing with various ciphers and compression settings and then some more tweaking and testing i just went and replaced the server with something more powerful.

I remember you having said somewhere that Oracle Solaris' ZFS encryption was faster than OpenZFS', does that still hold true now that encryption in OpenZFS has matured?
 

gea

Well-Known Member
Dec 31, 2010
2,870
1,013
113
DE
I have only compared ZFS and SMB performance where Solaris was faster, not encryption. In the last free OpenSolaris that is base of Solaris 11 and Open-ZFS, encryption was nearly ready. Oracle finished it in 2010, on Open-ZFS this was finished begin of last year based on the same.

It is known that disk encryption is faster than ZFS encryption. But as this is quite new there is room for improvements like the recent integration of CPU accelerated Raid-Z. Compared to ext4, the next performance disadvantage is the additional processing of realtime checksums and copy on write. While this is a huge security plus it costs performance.

ZFS encryption is fast enough for a medium class 10G filer (say 300-500MB/s). For 1000 MB/s or more you need a quite fast CPU. What may be a problem is the quite slow sync encryption with very small datablocks. If you want more than say 200 MB/s sync write you may need to disable encryption.

Without sync (and propably you have not used sync write on ext4), you are at >900 MB/s encrypted write on the Xeon and > 1700 MB/s on the AMD with a raid-Z of only three Optane (two datadisks).

Main problem now with "free" homeuse of Solaris is the freeze on 11.4 with rolling updates via service releases. They are guaranteed up to 2036 but only available with a subscription. Without a subscription Solaris is sadly more and more a problem as there are no new features and bugs or interoperability problems not get fixed in the free version.
 
Last edited:

Lahey

New Member
Nov 28, 2020
2
0
1
Yeah those were the figures i was seeing as well on testpools with 4-5 disks, where aes-192-gcm without compression (and sync=0) seemed to be the sweet spot, but the Xeon-D CPU was at its limit. Sequential writes of local files to the pool were fast but handling NFS/SMB traffic on top was too much (disks waiting for data), now on a modest E5-2620v3 it's very performant and stable. I will add more pools to test how far i can saturate the 10GbE links with that CPU. Who knew how much fun benchmarking can be :)

I'm glad to see how good the OpenSolaris variants are doing, now with native zfs encryption there aren't that many reasons to use Oracle Solaris (except for keeping Larry in yachts), especially not for homeuse i agree.