ZFS does Copy On Write. This means that it never overwrites data already on disk but always writes modified datablocks newly. The former datablocks are either marked as overwriteable for next writes or blocked by a snap. This means that ZFS snaps never write any data and are created nearly without delay as they only initiate a overwrite protection of data already on disk. Main advantage of Copy on Write is that this makes ZFS crash resistent as a power outage during write never corrupts the filesystem like with other filesystems. Disadvantage is a higher fragmentation and that data is often divided in more smaller datablocks. IO random performance of storage is then more relevant than pure sequential performance.
ZFS uses most of RAM and only RAM for read/write caching (You can extend RAM readcaching by a NVMe that makes cache also persistent). The write caching is mostly there to avoid small and slow random writes. The readcache works on a read most/ read last strategy but only for small io and metadata. It does not improve large sequential reads so if you need 2GB/s the pool must deliver this (and the network) on read and write - not trivial but achievable on a higher end server without sync write and encryption, see my tests with an Epyc system, https://www.napp-it.org/doc/downloads/epyc_performance.pdf
Regarding mechanical disks
They rate between 100 and 300 MB/s. On a mixed load I would count 150MB/s each. In a striped raid ex raid-0 you can theoretically double that value. If you stripe more disks the improvement is not linear more x1.5 with each added disk. 2GB/s stable read/write is hardly achievable with disks, more with SSD. I would use fast 12G SAS SSDs optionally in a dualpath 2x12G setup (Easier to scale than NVMe)
Last edited:
