Based on a current support discussion I made some tests with readcache disabled and some disks
to check effect of readcache (Cache for Metadate and/or Data) and disk iops.
For these tests I used filebench/ singlestreamread (napp-it menu Pools > Benchmark > Filebench):
Caches can be enabled/disabled in napp-it menu Pools > Pri/Sec cache.
Filer with a 60% fillrate, 23% fragmentation, 2 x 6 disk raid-z2 vdev of HGST HUS724040AL,
filer under decent load from students during test. The term mb/s in filebench means MegaByte/s.
all caches on
Now values of single disks
A singlestreamread on a single disk HGST HE8 Ultrastar
and a single Intel S3610-480 (all caches off)
Even the Re4 enterprise is not really good with 15 MB/s
Writevalues:
same WD Green with a singlestreamwrite and sync disabled (write cache on)
With the WD green, results with 3,1 MB/s are dramatically bad while results with the newer HGST 8TB disk and 85MB/s or an Intel SSD S3610 and 295 MB/s are ok. I am as well astonished about this very bad values on a slow disk with low iops values and cache disabled as mostly you do tests with all caches on to check overall system performance.
On writes, you always use the write cache that optimizes writes so default write values are much better. With sync=on you see similar weak write values as well with low iops disks. On a regular load, ZFS will hide these bad behaviour mostly with the readcache. So the result: avoid low iops disks and use as much RAM as possible for readcache.
to check effect of readcache (Cache for Metadate and/or Data) and disk iops.
For these tests I used filebench/ singlestreamread (napp-it menu Pools > Benchmark > Filebench):
Caches can be enabled/disabled in napp-it menu Pools > Pri/Sec cache.
Filer with a 60% fillrate, 23% fragmentation, 2 x 6 disk raid-z2 vdev of HGST HUS724040AL,
filer under decent load from students during test. The term mb/s in filebench means MegaByte/s.
all caches on
all caches off
read drops to 458 MB/s
Now values of single disks
A singlestreamread on a single disk HGST HE8 Ultrastar
the same but with metadata caching on (data caching off)
helps a little. With higher fragmentation, the difference may become bigger
and a single Intel S3610-480 (all caches off)
and finally a very old and slow WD green 1TB (all caches off) with 3,1 MByte/s
another low rpm disk (WD Re4 2TB enterprise)
Scary how bad the Greens are!!
Even the Re4 enterprise is not really good with 15 MB/s
Writevalues:
same WD Green with a singlestreamwrite and sync disabled (write cache on)
same WD Green with a singlestreamwrite and sync set to always (write cache off for sync write)
The result clearly shows. that ZFS performance especially on read highly depends on its caches, propably more than on other filesystems. So especially with very low RAM and disks with very low iops especially read performance of ZFS seems really bad. The faster the disks the better the values. With readcache performance can increase dramatically. High iops is a key value for performance.
With the WD green, results with 3,1 MB/s are dramatically bad while results with the newer HGST 8TB disk and 85MB/s or an Intel SSD S3610 and 295 MB/s are ok. I am as well astonished about this very bad values on a slow disk with low iops values and cache disabled as mostly you do tests with all caches on to check overall system performance.
On writes, you always use the write cache that optimizes writes so default write values are much better. With sync=on you see similar weak write values as well with low iops disks. On a regular load, ZFS will hide these bad behaviour mostly with the readcache. So the result: avoid low iops disks and use as much RAM as possible for readcache.
Last edited:
