Writes faster then reads on Mirror'ed pool

[Rand__]

I am playing around with pool design (device wise) and FreeNas/Napp-it.

Deployed the Napp-it template today and imported my Freenas 10 pool. Started some basic benchmarking using dd (since thats the easiest to replicate on FreeNas).

I removed compression & caching from the pool and got this:

Memory size: 3072 Megabytes

write 40.96 GB via dd, please wait...
time dd if=/dev/zero of=/intel/dd.tst bs=2048000 count=20000

20000+0 records in
20000+0 records out
40960000000 bytes transferred in 61.650757 secs (664387621 bytes/sec)

real     1:01.6
user        0.0
sys        10.0

40.96 GB in 61.6s = 664.94 MB/s Write

wait 40 s
read 40.96 GB via dd, please wait...
time dd if=/intel/dd.tst of=/dev/null bs=2048000

20000+0 records in
20000+0 records out
40960000000 bytes transferred in 112.303031 secs (364727468 bytes/sec)

real     1:52.3
user        0.0
sys        11.2

40.96 GB in 112.3s = 364.74 MB/s Read

The Pool is a striped mirror of 2x2Intel S3700 400GB, 1x2Samsung PM863 (960GB) and 1x2 HGST HUSML4040ASS600 400GB.

Can anyone explain why read is slower than write?
I checked CPU utilization but thats fine...

 

[Markus]

How did you deactivated the caching?

Regards
Markus

 

[gigatexal]

Dd is not a tool to make any definitive conclusions from. Use fio

Fio - Flexible I/O Tester Synthetic Benchmark | StorageReview.com - Storage Reviews

 

[Rand__]

How did you deactivated the caching?

Via the gui (Pools)


I used dd as thats nativly available on FN as well, and while it might not be a good tool it should return consistent results, should it not?
But will look at FIO, no problem

 

[whitey]

oflag=direct will help on dd runs to avoid using the host's page cache for writes and write directly to disk

 

[gea]

The main problem
- ZFS does everything to be slow, more cpu load or data due checksums and more fragmentation due CopyOnWrite (compared to other filesystems). You do not get the ZFS data security for free.

ZFS compensate this with advanced cache strategies. If you disable all read caching, you fall back to pure disk performance on reads. On writes you use write caching. ZFS use between 10% and 4 GB RAM per default as write cache to convert many slow small random writes to a single fast large sequential write (up to about 5s of data).

This may be the reason why write is faster than read when disabling read caches (or low RAM what means the same).

 

[Rand__]

oflag=direct will help on dd runs to avoid using the host's page cache for writes and write directly to disk

I have

Each FLAG symbol may be:

  append    append mode (makes sense only for output; conv=notrunc suggested)
  dsync     use synchronized I/O for data
  sync      likewise, but also for metadata
  fullblock  accumulate full blocks of input (iflag only)
  nonblock  use non-blocking I/O
  nocache   Request to drop cache.  See also oflag=sync
  noctty    do not assign controlling terminal from file
  nofollow  do not follow symlinks
  nolinks   fail if multiply-linked
  count_bytes  treat 'count=N' as a byte count (iflag only)
  skip_bytes  treat 'skip=N' as a byte count (iflag only)
  seek_bytes  treat 'seek=N' as a byte count (oflag only)

So I chose nocache
Not much of a difference:

time dd if=/dev/zero of=/intel/dd.tst bs=2048000 count=100000 oflag=nocache
100000+0 records in
100000+0 records out
204800000000 bytes (205 GB, 191 GiB) copied, 292.483 s, 700 MB/s

real    4m52.485s
user    0m0.333s
sys     1m2.335s

root@nappIT1:/root# time dd of=/dev/null if=/intel/dd.tst bs=2048000 count=100000 oflag=nocache
100000+0 records in
100000+0 records out
204800000000 bytes (205 GB, 191 GiB) copied, 656.529 s, 312 MB/s

real    10m56.532s
user    0m0.274s
sys     1m0.830s

ZFS compensate this with advanced cache strategies. If you disable all read caching, you fall back to pure disk performance on reads. On writes you use write caching. ZFS use between 10% and 4 GB RAM per default as write cache to convert many slow small random writes to a single fast large sequential write (up to about 5s of data).

So how do i turn of write caching? The nocache option did not really do much

And why are there so few people with this issue? You recommend to turn of caching in Napp-it (gui), so basically all users should experience this. But I couldn't find much on similar topics...

 

[gea]

No, disabling readcaches on ZFS is a very bad idea for regular operation.
Its only ok to compare raw read disk performance between two settings.

With disks read should be similar to write (SSD is different, read should be faster there)

Tuning write throttle is different, I would simply keep defaults
Adam Leventhal's blog » Tuning the OpenZFS write throttle

 

[Rand__]

Yes o/c, I meant for benchmarking only But nevertheless it is odd that write is faster than read speed, hence my question.
So no way to turn of write cache to get actual disk speed on writes too?

 

[T_Minus]

Write cache I think you can limit by setting the max RAM usage and then writing more than that... iE: set it to 4gb and then write 16GB.

 

[Rand__]

Yes I thought that helped too, that's why I ran a run with 200GB write size - same write speed (see reply to @whitey in #7).

Edit: Numbers look a bit better now (800 to 400) , realized the controller was in an PCIe2x4 slot instead of x8.
Didn't change the imbalance though

 

[gea]

You may tune the size of the write cache/ write throtteling
with the zfs_dirty_data_max setting
Tuning the OpenZFS write throttle | Delphix