optimize zfs for seq reads

ekke · Feb 26, 2024

gb00s said:
That special devices can't be removed is so wrong as the earth is a disc. Special devs in mirror and same ashift can be removed. I had much better performance and sec with specials devs in Raid10 than a x-way mirror w/Optane 4800x and DCPMM.

Edit:

What if you get rid of all the extra devs, specially the special devs?

Hm, so special vdevs can be removed ? (if no small files stored on them)But I do store small files there.

well two mirrors are faster.

I got total of 3 drives used in diffeent namespaces. done have four drives avaible
but I dont see how special vdev can releate to slow reads?

ekke · Feb 26, 2024

gea said:
ZFS Arc or L2Arc defaults for read ahead are quite optimal for all sort of workloads. You can modify for a certain workload but this can decrease performance on other workloads. In general these settings can increase performance for a few percent only.

If you say that your read performance is 30 MB/s then this is bad. If you can tune to increase 30% then you are at 40 MB/s what is bad as well. I do not asume that 30% is possible.

I would first look why this is so low. Maybe one or some disks are bad. Check iostat on load if all disks have same load. You can also build a pool per disk and compare performance. When ZFS writes to a pool it spreads data over the pool ao achieve a constant performance with many files and users. ZFS does not optimize for a single user with a large file. The datasheet disk values are lab values when you fill a disk track ba track. This is not the case with Raid or ZFS. But in general you should achieve at least 80 MB/s per disk.

Mostly sequential or large fileread performance depend on pool layout like n-way mirrors where ZFS can read from all mirror disks concurrently (a mirror can have 2x single disk performance)) or n-vdev Raid-Z and raw single single disk performance. A 1M recsice can help a little over a default 128K with the price of inefficiency on small files.

Arc and L2Arc tunings can help a little on small io but cache is for most/last read ZFS datablocks not files. Mostly performance is bad for small io. L2Arc can help a little as it is persistent or on low RAM situations. If you have enough RAM or not a large number of users with volatile files ex a university mailserver, forget L2Arc. A special vdev mirror is a better option. If you set a small blocksize for example of 64GB with a recsize of 128K+, then all files with size up to 64K (nearly all "office files") are stored on the special vdev, not only metadata. If you set 64K and a recsize 64K, all files of this filesystem are on the special vdev so you can set per filesystem workload.

But first, investigate why average single disk performance is so low.

as you can see from the images, Im using 8 disks in a raidz2, reads via zpool iostat and iostat loads are eqaully spread over the drives.

arc/l2arc tuning wont help here, I was looking at mostly zfetch, the readahead value see if that could increase the reads.

Yes I got small blocksize set to 64Kb for the pool default, then tweak for the zfs's.

generally good performance exect the reads here that are so slow compared to writes. Ill do some more testing during next weekend if I have the time.

This is also just my home server, so its not a "real" issue but labbing a testing for fun. although I got all my personal data on there.

ekke · Feb 26, 2024

gb00s said:
That special devices can't be removed is so wrong as the earth is a disc. Special devs in mirror and same ashift can be removed. I had much better performance and sec with specials devs in Raid10 than a x-way mirror w/Optane 4800x and DCPM (don't ask me why and how).

Edit:

What if you get rid of all the extra devs, specially the special devs?

you got more performance due to more vdevs.

BackupProphet · Feb 26, 2024

A 3way mirror for special vdev will have 6 copies of your metadata. Unless you have redundant_metadata=none. In my opinion that is way overkill.

SnJ9MX · Feb 26, 2024

ekke said:
Hm, so special vdevs can be removed ? (if no small files stored on them)But I do store small files there.

well two mirrors are faster.

I got total of 3 drives used in diffeent namespaces. done have four drives avaible
but I dont see how special vdev can releate to slow reads?

yes they can. the small files will be written back to the underlying vdevs

gea · Feb 26, 2024

SnJ9MX said:
yes they can. the small files will be written back to the underlying vdevs

- If you remove a special vdev, data is moved to the pool.
- A vdev lost is otherwise a pool lost as data is not cached there but stored.
- a 3way mirror is not perfect for read performance but allow 2 disks to fail.

thulle · Feb 26, 2024

ekke said:
echo 67108864 > /sys/module/zfs/parameters/zfetch_max_distance
didnt increase the seq reads.

Isn't that the default value?

Code:

# uname -a
FreeBSD 13.1-RELEASE-p7
# zfs -V
zfs-2.1.11-1
zfs-kmod-v2023072100-zfs_0eb787a7e
# sysctl -a|grep zfetch
vfs.zfs.zfetch.max_idistance: 67108864
vfs.zfs.zfetch.max_distance: 67108864


$ uname -a
Linux 6.7.6-gentoo-x86_64
# zfs -V
zfs-2.2.3-r0-gentoo
zfs-kmod-2.2.3-r0-gentoo
$ cat /sys/module/zfs/parameters/zfetch_max_distance
67108864

ekke · Feb 27, 2024

Code:

pve# zpool get all tank
NAME  PROPERTY                       VALUE                          SOURCE
tank  size                           71.9T                          -
tank  capacity                       48%                            -
tank  altroot                        -                              default
tank  health                         ONLINE                         -
tank  guid                           8026931783135761224            -
tank  version                        -                              default
tank  bootfs                         -                              default
tank  delegation                     on                             default
tank  autoreplace                    off                            default
tank  cachefile                      none                           local
tank  failmode                       wait                           default
tank  listsnapshots                  off                            default
tank  autoexpand                     off                            default
tank  dedupratio                     1.00x                          -
tank  free                           37.0T                          -
tank  allocated                      34.9T                          -
tank  readonly                       off                            -
tank  ashift                         12                             local
tank  comment                        -                              default
tank  expandsize                     -                              -
tank  freeing                        0                              -
tank  fragmentation                  0%                             -
tank  leaked                         0                              -
tank  multihost                      off                            default
tank  checkpoint                     -                              -
tank  load_guid                      5866013282328893016            -
tank  autotrim                       off                            default
tank  compatibility                  off                            default
tank  bcloneused                     0                              -
tank  bclonesaved                    0                              -
tank  bcloneratio                    1.00x                          -
tank  feature@async_destroy          enabled                        local
tank  feature@empty_bpobj            active                         local
tank  feature@lz4_compress           active                         local
tank  feature@multi_vdev_crash_dump  enabled                        local
tank  feature@spacemap_histogram     active                         local
tank  feature@enabled_txg            active                         local
tank  feature@hole_birth             active                         local
tank  feature@extensible_dataset     active                         local
tank  feature@embedded_data          active                         local
tank  feature@bookmarks              enabled                        local
tank  feature@filesystem_limits      enabled                        local
tank  feature@large_blocks           active                         local
tank  feature@large_dnode            enabled                        local
tank  feature@sha512                 enabled                        local
tank  feature@skein                  enabled                        local
tank  feature@edonr                  enabled                        local
tank  feature@userobj_accounting     active                         local
tank  feature@encryption             enabled                        local
tank  feature@project_quota          active                         local
tank  feature@device_removal         enabled                        local
tank  feature@obsolete_counts        enabled                        local
tank  feature@zpool_checkpoint       enabled                        local
tank  feature@spacemap_v2            active                         local
tank  feature@allocation_classes     active                         local
tank  feature@resilver_defer         enabled                        local
tank  feature@bookmark_v2            enabled                        local
tank  feature@redaction_bookmarks    enabled                        local
tank  feature@redacted_datasets      enabled                        local
tank  feature@bookmark_written       enabled                        local
tank  feature@log_spacemap           active                         local
tank  feature@livelist               enabled                        local
tank  feature@device_rebuild         enabled                        local
tank  feature@zstd_compress          enabled                        local
tank  feature@draid                  enabled                        local
tank  feature@zilsaxattr             active                         local
tank  feature@head_errlog            active                         local
tank  feature@blake3                 enabled                        local
tank  feature@block_cloning          enabled                        local
tank  feature@vdev_zaps_v2           active                         local
pve# zfs get all tank/media
NAME        PROPERTY              VALUE                      SOURCE
tank/media  type                  filesystem                 -
tank/media  creation              lör feb 10 12:39 2024     -
tank/media  used                  18.4T                      -
tank/media  available             25.7T                      -
tank/media  referenced            17.9T                      -
tank/media  compressratio         1.01x                      -
tank/media  mounted               yes                        -
tank/media  quota                 none                       default
tank/media  reservation           none                       default
tank/media  recordsize            1M                         local
tank/media  mountpoint            /media                     local
tank/media  sharenfs              off                        default
tank/media  checksum              on                         default
tank/media  compression           lz4                        local
tank/media  atime                 off                        inherited from tank
tank/media  devices               on                         default
tank/media  exec                  on                         default
tank/media  setuid                on                         default
tank/media  readonly              off                        default
tank/media  zoned                 off                        default
tank/media  snapdir               hidden                     default
tank/media  aclmode               discard                    default
tank/media  aclinherit            restricted                 default
tank/media  createtxg             569222                     -
tank/media  canmount              on                         default
tank/media  xattr                 on                         default
tank/media  copies                1                          default
tank/media  version               5                          -
tank/media  utf8only              off                        -
tank/media  normalization         none                       -
tank/media  casesensitivity       insensitive                -
tank/media  vscan                 off                        default
tank/media  nbmand                off                        default
tank/media  sharesmb              off                        default
tank/media  refquota              none                       default
tank/media  refreservation        none                       default
tank/media  guid                  12516367741825142107       -
tank/media  primarycache          all                        default
tank/media  secondarycache        all                        default
tank/media  usedbysnapshots       485G                       -
tank/media  usedbydataset         17.9T                      -
tank/media  usedbychildren        0B                         -
tank/media  usedbyrefreservation  0B                         -
tank/media  logbias               latency                    default
tank/media  objsetid              1801                       -
tank/media  dedup                 off                        default
tank/media  mlslabel              none                       default
tank/media  sync                  standard                   default
tank/media  dnodesize             legacy                     default
tank/media  refcompressratio      1.01x                      -
tank/media  written               7.45T                      -
tank/media  logicalused           19.0T                      -
tank/media  logicalreferenced     18.5T                      -
tank/media  volmode               default                    default
tank/media  filesystem_limit      none                       default
tank/media  snapshot_limit        none                       default
tank/media  filesystem_count      none                       default
tank/media  snapshot_count        none                       default
tank/media  snapdev               hidden                     default
tank/media  acltype               off                        default
tank/media  context               none                       default
tank/media  fscontext             none                       default
tank/media  defcontext            none                       default
tank/media  rootcontext           none                       default
tank/media  relatime              on                         inherited from tank
tank/media  redundant_metadata    all                        default
tank/media  overlay               on                         default
tank/media  encryption            off                        default
tank/media  keylocation           none                       default
tank/media  keyformat             none                       default
tank/media  pbkdf2iters           0                          default
tank/media  special_small_blocks  64K                        inherited from tank
tank/media  snapshots_changed     sön feb 11  4:09:14 2024  -
pve#

ekke · Feb 27, 2024

thulle said:

Isn't that the default value?

Code:

# uname -a
FreeBSD 13.1-RELEASE-p7
# zfs -V
zfs-2.1.11-1
zfs-kmod-v2023072100-zfs_0eb787a7e
# sysctl -a|grep zfetch
vfs.zfs.zfetch.max_idistance: 67108864
vfs.zfs.zfetch.max_distance: 67108864


$ uname -a
Linux 6.7.6-gentoo-x86_64
# zfs -V
zfs-2.2.3-r0-gentoo
zfs-kmod-2.2.3-r0-gentoo
$ cat /sys/module/zfs/parameters/zfetch_max_distance
67108864

These are all my zfetch values:

pve# cat /sys/module/zfs/parameters/zfetch*
67108864
67108864
2
8
4194304
1
pve#

ekke · Feb 27, 2024

I have turn off all torrents etc, doing reads.

when I stream a film over lan it buffers now and then! something is really off here.

iostat shows sub 1% util on all drives

ekke · Feb 27, 2024

Code:

pve# zpool iostat tank -vw

tank                                                total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0   175K   400K  5.96M   375K   192K      0      0
511ns                                                  0      0      0      0  6.58M  2.07M  53.0M  6.14M   892K      0      0
1us                                                    0      0      0      0  3.43M   233K  15.9M  10.6M   416K      0      0
2us                                                    0      0      0      0   690K  34.9K  1020K  3.95M  18.7K      0      0
4us                                                    0      0      0      0  17.5K  9.77K  60.0K   526K  13.0K      0      0
8us                                                    0     28      0    426  2.44K  2.82K  64.9K   735K  16.2K      0      0
16us                                                   2  1.95M      5  4.38M  7.45K  1.90K   102K  1.31M  38.3K      0      0
32us                                                 110  4.65M    132  7.32M  13.0K    762   187K  1.69M  75.0K      0      0
65us                                               11.8M  3.89M  11.9M  2.94M    308     60  79.1K  1.97M   132K      0      0
131us                                              20.7M  3.24M  20.6M  1.70M    205     82   181K  2.22M   108K      0      0
262us                                              24.0M  2.17M  24.3M  1004K    617    158   391K  1.53M  29.4K      0      0
524us                                              21.7M  2.98M  21.8M  2.90M  1.12K    263   235K  1017K  2.21K      0      0
1ms                                                2.41M  1.34M  2.22M  1.26M  1.07K    406   100K   943K  1.05K      0      0
2ms                                                 451K  1.18M   478K  1.31M    573    651  95.2K  1.08M  1.83K      0      0
4ms                                                 608K  1.64M   750K  2.54M    839    967   123K  1.57M  2.43K      0      0
8ms                                                1.75M  4.08M  2.01M  7.45M    939  1.12K   170K  2.48M  1.80K      0      0
16ms                                               4.21M  7.25M  4.29M  8.34M  1.17K  1.26K   194K  2.91M    389      0      0
33ms                                               2.75M  7.72M  2.65M  5.08M    924  1.30K   138K  3.32M    700      0      0
67ms                                                302K  5.26M   246K  2.78M    512    727  48.1K  2.18M  1.26K      0      0
134ms                                               106K  2.07M  62.2K   999K  1.75K    713  29.9K   607K  2.43K      0      0
268ms                                              67.0K   574K  47.6K   181K    649    443  11.0K   252K  5.11K      0      0
536ms                                              45.5K   188K  31.6K  36.7K    249      0  2.86K  99.7K  9.92K      0      0
1s                                                 29.8K  28.5K  10.5K  12.1K      2      0    496  5.55K  18.6K      0      0
2s                                                 37.7K     83      0      0      0      0      0      0  37.7K      0      0
4s                                                 71.9K      0      0      0      0      0      0      0  71.9K      0      0
8s                                                  171K      0      0      0      0      0      0      0   171K      0      0
17s                                                 249K      0      0      0      0      0      0      0   249K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

raidz2-0                                            total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0   172K   324K  5.96M    453      0      0      0
511ns                                                  0      0      0      0  6.55M  1.49M  53.0M  3.32M    139      0      0
1us                                                    0      0      0      0  3.40M  88.1K  15.9M  8.30M    347      0      0
2us                                                    0      0      0      0   685K  4.35K  1019K  3.21M    106      0      0
4us                                                    0      0      0      0  17.3K  5.30K  59.9K   142K     16      0      0
8us                                                    0      0      0      0  2.41K  1.88K  64.8K  46.9K      0      0      0
16us                                                   0      0      0      0  7.38K  1.12K   102K  74.9K      1      0      0
32us                                                   9      0     10      0  12.9K    165   187K   121K      1      0      0
65us                                               11.7M  67.5K  11.8M  70.5K    284     10  79.0K   163K      3      0      0
131us                                              20.2M   112K  20.1M   110K    181      1   181K   273K      2      0      0
262us                                              23.1M   250K  23.3M   299K    566      0   391K   392K      2      0      0
524us                                              21.2M  2.34M  21.4M  2.40M  1.11K      0   235K   622K      5      0      0
1ms                                                2.39M  1.15M  2.21M  1.25M  1.07K      0   100K   819K     12      0      0
2ms                                                 449K  1.11M   478K  1.29M    573      0  95.2K  1.04M     30      0      0
4ms                                                 606K  1.60M   750K  2.54M    839      0   123K  1.54M     56      0      0
8ms                                                1.74M  4.02M  2.01M  7.45M    939      0   170K  2.43M    103      0      0
16ms                                               4.21M  7.09M  4.29M  8.34M  1.17K      0   194K  2.75M    299      0      0
33ms                                               2.75M  7.06M  2.65M  5.07M    924      0   138K  2.68M    700      0      0
67ms                                                302K  5.21M   246K  2.78M    512      0  48.1K  2.13M  1.26K      0      0
134ms                                               106K  2.04M  62.2K   998K  1.75K      0  29.9K   579K  2.43K      0      0
268ms                                              67.0K   557K  47.6K   181K    649      0  11.0K   235K  5.11K      0      0
536ms                                              45.5K   185K  31.6K  36.7K    249      0  2.86K  97.5K  9.92K      0      0
1s                                                 29.8K  28.5K  10.5K  12.1K      2      0    496  5.54K  18.6K      0      0
2s                                                 37.7K     83      0      0      0      0      0      0  37.7K      0      0
4s                                                 71.9K      0      0      0      0      0      0      0  71.9K      0      0
8s                                                  171K      0      0      0      0      0      0      0   171K      0      0
17s                                                 249K      0      0      0      0      0      0      0   249K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

scsi-35000cca2a23c5008                              total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  24.3K  13.9K   992K     93      0      0      0
511ns                                                  0      0      0      0   902K   201K  6.92M   466K     20      0      0
1us                                                    0      0      0      0   402K  27.6K  1.51M  1.06M     42      0      0
2us                                                    0      0      0      0  53.5K  1.52K  86.1K   504K     14      0      0
4us                                                    0      0      0      0  1.54K     78  6.78K  19.0K      1      0      0
8us                                                    0      0      0      0    305     14  7.75K  6.40K      0      0      0
16us                                                   0      0      0      0    974     54  12.4K  9.87K      0      0      0
32us                                                   1      0      1      0  1.54K     11  22.0K  15.8K      1      0      0
65us                                               1.47M  8.54K  1.48M  8.94K     35      0  9.84K  21.2K      0      0      0
131us                                              2.52M  14.2K  2.51M  14.0K     20      0  22.8K  35.2K      0      0      0
262us                                              2.86M  31.2K  2.89M  37.6K     68      0  49.0K  50.1K      0      0      0
524us                                              2.68M   305K  2.70M   314K    148      0  29.4K  83.5K      0      0      0
1ms                                                 306K   145K   282K   160K    126      0  12.6K   114K      2      0      0
2ms                                                55.5K   147K  58.9K   174K     66      0  11.8K   149K      4      0      0
4ms                                                75.6K   226K  94.3K   397K    104      0  15.1K   223K      8      0      0
8ms                                                 227K   589K   261K   992K     94      0  21.7K   350K     14      0      0
16ms                                                546K  1.01M   555K  1.08M    167      0  24.5K   382K     40      0      0
33ms                                                342K   972K   329K   668K    100      0  16.9K   316K     74      0      0
67ms                                               36.4K   589K  29.4K   342K     55      0  5.64K   191K    134      0      0
134ms                                              12.2K   210K  7.75K   112K     39      0  2.78K  55.5K    320      0      0
268ms                                              7.27K  59.1K  5.64K  19.8K     12      0    756  25.3K    631      0      0
536ms                                              5.01K  19.1K  3.45K  4.26K      5      0    187  9.25K  1.32K      0      0
1s                                                 3.36K  2.21K    990    944      0      0     57    395  2.35K      0      0
2s                                                 4.71K      5      0      0      0      0      0      0  4.71K      0      0
4s                                                 9.03K      0      0      0      0      0      0      0  9.02K      0      0
8s                                                 21.2K      0      0      0      0      0      0      0  21.3K      0      0
17s                                                30.9K      0      0      0      0      0      0      0  30.8K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

scsi-35000cca2a23c3fd4                              total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  21.4K  33.0K   671K     56      0      0      0
511ns                                                  0      0      0      0   780K   202K  6.23M   458K     17      0      0
1us                                                    0      0      0      0   465K  8.05K  2.44M  1.14M     47      0      0
2us                                                    0      0      0      0   115K    391   167K   361K     11      0      0
4us                                                    0      0      0      0  2.84K     27  8.26K  17.3K      1      0      0
8us                                                    0      0      0      0    323     20  8.36K  5.76K      0      0      0
16us                                                   0      0      0      0    939     60  12.8K  9.50K      0      0      0
32us                                                   4      0      4      0  1.62K      9  25.0K  15.5K      0      0      0
65us                                               1.47M  8.51K  1.48M  8.86K     33      1  10.0K  20.8K      1      0      0
131us                                              2.53M  14.2K  2.52M  14.1K     27      0  22.6K  35.2K      0      0      0
262us                                              2.91M  31.6K  2.94M  37.7K     61      0  49.0K  50.0K      1      0      0
524us                                              2.63M   308K  2.66M   317K    112      0  29.2K  81.7K      0      0      0
1ms                                                 306K   141K   282K   155K    127      0  12.6K   111K      2      0      0
2ms                                                56.1K   147K  59.4K   172K     68      0  11.8K   145K      5      0      0
4ms                                                75.5K   218K  94.0K   371K     95      0  15.3K   217K      8      0      0
8ms                                                 226K   556K   260K   977K    108      0  21.6K   339K     17      0      0
16ms                                                544K   989K   554K  1.06M    143      0  24.4K   377K     41      0      0
33ms                                                343K   958K   330K   667K    115      0  17.1K   332K     83      0      0
67ms                                               37.2K   616K  30.1K   346K     62      0  5.73K   212K    171      0      0
134ms                                              12.7K   223K  7.86K   117K     71      0  3.35K  59.1K    281      0      0
268ms                                              7.58K  64.3K  5.82K  21.3K     20      0    857  27.5K    616      0      0
536ms                                              4.99K  20.1K  3.53K  4.37K      5      0    204  9.81K  1.22K      0      0
1s                                                 3.45K  2.58K  1.08K  1.09K      0      0     48    431  2.30K      0      0
2s                                                 4.62K      3      0      0      0      0      0      0  4.63K      0      0
4s                                                 9.11K      0      0      0      0      0      0      0  9.11K      0      0
8s                                                 21.5K      0      0      0      0      0      0      0  21.5K      0      0
17s                                                31.2K      0      0      0      0      0      0      0  31.1K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

scsi-35000cca2a23c505c                              total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  21.6K  42.3K   811K     65      0      0      0
511ns                                                  0      0      0      0   902K   193K  7.08M   390K     19      0      0
1us                                                    0      0      0      0   406K  8.13K  1.52M  1009K     40      0      0
2us                                                    0      0      0      0  53.9K    465  85.7K   454K     13      0      0
4us                                                    0      0      0      0  1.50K     33  6.80K  18.3K      5      0      0
8us                                                    0      0      0      0    279     22  7.85K  6.03K      0      0      0
16us                                                   0      0      0      0    894     72  12.5K  9.23K      0      0      0
32us                                                   0      0      0      0  1.55K     14  22.3K  15.0K      0      0      0
65us                                               1.46M  8.83K  1.48M  9.19K     30      2  9.72K  19.9K      1      0      0
131us                                              2.52M  13.9K  2.51M  13.7K     17      1  22.5K  33.3K      0      0      0
262us                                              2.86M  31.0K  2.89M  37.0K     80      0  48.9K  48.2K      0      0      0
524us                                              2.68M   309K  2.70M   317K    161      0  29.6K  74.9K      0      0      0
1ms                                                 306K   137K   282K   149K    137      0  12.3K  96.5K      1      0      0
2ms                                                56.3K   141K  60.3K   163K     68      0  11.8K   123K      2      0      0
4ms                                                75.6K   197K  93.8K   296K    116      0  15.0K   180K      5      0      0
8ms                                                 225K   486K   260K   915K    102      0  21.0K   285K     18      0      0
16ms                                                541K   852K   553K  1.05M    163      0  23.7K   327K     36      0      0
33ms                                                345K   858K   334K   643K    134      0  16.3K   338K     96      0      0
67ms                                               36.5K   702K  31.0K   360K     78      0  5.18K   331K    172      0      0
134ms                                              14.5K   301K  7.66K   127K    957      0  4.96K  88.1K    319      0      0
268ms                                              10.3K  71.5K  6.05K  22.2K    358      0  2.88K  32.1K    673      0      0
536ms                                              6.77K  26.5K  4.21K  4.79K    139      0    946  14.5K  1.23K      0      0
1s                                                 4.06K  4.50K  1.62K  1.90K      2      0     95    976  2.31K      0      0
2s                                                 4.71K     16      0      0      0      0      0      0  4.72K      0      0
4s                                                 9.16K      0      0      0      0      0      0      0  9.17K      0      0
8s                                                 21.7K      0      0      0      0      0      0      0  21.7K      0      0
17s                                                31.0K      0      0      0      0      0      0      0  31.0K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

scsi-35000cca2a23c4558                              total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  15.6K  50.2K   607K     46      0      0      0
511ns                                                  0      0      0      0   754K   184K  6.22M   397K     16      0      0
1us                                                    0      0      0      0   490K  8.45K  2.49M   953K     40      0      0
2us                                                    0      0      0      0   121K    470   172K   324K     20      0      0
4us                                                    0      0      0      0  2.86K     48  8.40K  16.3K      2      0      0
8us                                                    0      0      0      0    346     32  8.22K  5.26K      0      0      0
16us                                                   0      0      0      0    938     78  13.1K  8.61K      0      0      0
32us                                                   1      0      1      0  1.68K     17  24.8K  14.1K      0      0      0
65us                                               1.46M  8.08K  1.47M  8.42K     37      2  9.96K  18.8K      0      0      0
131us                                              2.53M  13.4K  2.52M  13.3K     25      0  22.9K  32.4K      0      0      0
262us                                              2.90M  31.2K  2.92M  37.0K     75      0  49.1K  46.3K      0      0      0
524us                                              2.63M   302K  2.65M   311K    156      0  29.7K  68.4K      0      0      0
1ms                                                 307K   137K   283K   148K    153      0  12.7K  83.6K      2      0      0
2ms                                                57.3K   135K  60.9K   153K     75      0  12.0K   107K      3      0      0
4ms                                                76.1K   177K  93.4K   268K     97      0  15.5K   154K      7      0      0
8ms                                                 219K   415K   254K   835K    124      0  21.1K   255K      5      0      0
16ms                                                516K   720K   530K  1013K    150      0  23.0K   297K     28      0      0
33ms                                                378K   757K   366K   611K    116      0  17.2K   350K     98      0      0
67ms                                               40.4K   779K  34.0K   387K    101      0  6.24K   425K    188      0      0
134ms                                              15.2K   367K  7.63K   144K    463      0  5.50K   107K    335      0      0
268ms                                              10.2K  82.8K  6.04K  25.4K    198      0  2.81K  33.9K    686      0      0
536ms                                              6.84K  29.7K  4.76K  5.04K     73      0    599  17.5K  1.16K      0      0
1s                                                 4.44K  6.32K  1.96K  2.66K      0      0     83  1.41K  2.37K      0      0
2s                                                 4.71K     31      0      0      0      0      0      0  4.72K      0      0
4s                                                 9.01K      0      0      0      0      0      0      0  9.02K      0      0
8s                                                 21.5K      0      0      0      0      0      0      0  21.5K      0      0
17s                                                31.0K      0      0      0      0      0      0      0  30.9K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

scsi-35000cca2a23c4e88                              total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  24.0K  55.7K   857K     48      0      0      0
511ns                                                  0      0      0      0   907K   179K  7.05M   421K     20      0      0
1us                                                    0      0      0      0   396K  8.04K  1.52M  1000K     51      0      0
2us                                                    0      0      0      0  54.3K    418  86.3K   460K      7      0      0
4us                                                    0      0      0      0  1.42K     46  6.63K  18.2K      2      0      0
8us                                                    0      0      0      0    250     76  8.04K  6.16K      0      0      0
16us                                                   0      0      0      0    925    108  12.5K  9.47K      0      0      0
32us                                                   1      0      2      0  1.53K     20  22.1K  15.2K      0      0      0
65us                                               1.47M  8.33K  1.48M  8.70K     42      0  9.73K  20.3K      1      0      0
131us                                              2.53M  13.8K  2.52M  13.6K     20      0  22.4K  34.2K      0      0      0
262us                                              2.85M  31.0K  2.89M  37.1K     80      0  48.4K  49.4K      0      0      0
524us                                              2.68M   306K  2.70M   314K    139      0  29.5K  77.5K      2      0      0
1ms                                                 308K   140K   284K   152K    126      0  12.5K   102K      1      0      0
2ms                                                56.3K   140K  60.1K   162K     64      0  11.9K   132K      4      0      0
4ms                                                76.0K   199K  93.7K   309K    106      0  15.6K   196K      8      0      0
8ms                                                 215K   504K   250K   971K    123      0  21.1K   309K     17      0      0
16ms                                                540K   897K   552K  1.02M    140      0  25.0K   357K     40      0      0
33ms                                                359K   923K   346K   654K    107      0  17.5K   358K     96      0      0
67ms                                               38.4K   672K  30.8K   354K     58      0  6.42K   263K    149      0      0
134ms                                              13.3K   249K  7.83K   125K     90      0  3.73K  66.9K    283      0      0
268ms                                              8.21K  70.6K  6.05K  23.8K     38      0  1.14K  29.2K    653      0      0
536ms                                              5.46K  23.4K  3.91K  4.65K     10      0    277  12.4K  1.20K      0      0
1s                                                 3.68K  3.54K  1.27K  1.53K      0      0     46    720  2.35K      0      0
2s                                                 4.69K      7      0      0      0      0      0      0  4.69K      0      0
4s                                                 9.08K      0      0      0      0      0      0      0  9.07K      0      0
8s                                                 21.6K      0      0      0      0      0      0      0  21.6K      0      0
17s                                                31.0K      0      0      0      0      0      0      0  30.9K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

scsi-35000cca2a23c3fe0                              total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  20.4K  62.1K   637K     39      0      0      0
511ns                                                  0      0      0      0   770K   173K  6.24M   440K     14      0      0
1us                                                    0      0      0      0   471K  8.23K  2.47M  1.12M     45      0      0
2us                                                    0      0      0      0   118K    361   169K   358K     13      0      0
4us                                                    0      0      0      0  2.79K    158  8.29K  17.2K      2      0      0
8us                                                    0      0      0      0    342    181  8.26K  5.54K      0      0      0
16us                                                   0      0      0      0   1000    102  13.2K  9.40K      0      0      0
32us                                                   0      0      0      0  1.67K     26  24.2K  15.2K      0      0      0
65us                                               1.47M  8.39K  1.49M  8.78K     36      1  9.64K  20.7K      0      0      0
131us                                              2.53M  13.7K  2.52M  13.5K     20      0  22.6K  34.5K      0      0      0
262us                                              2.91M  31.6K  2.94M  37.6K     64      0  48.9K  50.2K      1      0      0
524us                                              2.63M   306K  2.65M   315K    128      0  29.3K  80.5K      1      0      0
1ms                                                 307K   139K   283K   152K    144      0  12.4K   107K      2      0      0
2ms                                                56.1K   143K  59.5K   166K     69      0  11.9K   139K      3      0      0
4ms                                                75.6K   209K  93.1K   324K    120      0  15.3K   208K      6      0      0
8ms                                                 226K   534K   261K   998K    121      0  21.1K   324K      9      0      0
16ms                                                542K   947K   553K  1.05M    149      0  24.2K   366K     41      0      0
33ms                                                344K   944K   331K   658K    109      0  17.2K   348K     90      0      0
67ms                                               37.6K   648K  29.8K   351K     58      0  6.57K   236K    166      0      0
134ms                                              12.8K   233K  7.83K   121K     52      0  3.30K  62.3K    335      0      0
268ms                                              7.85K  66.6K  6.07K  21.8K      7      0    854  27.8K    666      0      0
536ms                                              5.20K  21.0K  3.64K  4.39K      1      0    234  10.7K  1.28K      0      0
1s                                                 3.45K  2.58K  1.10K  1.22K      0      0     53    370  2.27K      0      0
2s                                                 4.75K      6      0      0      0      0      0      0  4.76K      0      0
4s                                                 8.90K      0      0      0      0      0      0      0  8.89K      0      0
8s                                                 21.2K      0      0      0      0      0      0      0  21.3K      0      0
17s                                                31.4K      0      0      0      0      0      0      0  31.3K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

scsi-35000cca2a23c51b0                              total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  26.0K  40.3K   885K     59      0      0      0
511ns                                                  0      0      0      0   924K   190K  7.01M   404K     16      0      0
1us                                                    0      0      0      0   380K  9.62K  1.53M  1.03M     46      0      0
2us                                                    0      0      0      0  52.9K    380  87.0K   482K     16      0      0
4us                                                    0      0      0      0  1.47K  2.22K  6.73K  18.6K      2      0      0
8us                                                    0      0      0      0    271    501  8.06K  6.20K      0      0      0
16us                                                   0      0      0      0    940    573  12.5K  9.58K      0      0      0
32us                                                   1      0      1      0  1.58K     29  22.1K  15.5K      0      0      0
65us                                               1.47M  8.50K  1.48M  8.88K     34      3  10.1K  20.7K      0      0      0
131us                                              2.53M  14.1K  2.51M  13.9K     18      0  22.6K  34.9K      1      0      0
262us                                              2.86M  31.2K  2.89M  37.3K     73      0  49.0K  49.7K      0      0      0
524us                                              2.68M   287K  2.70M   296K    162      0  29.3K  79.2K      1      0      0
1ms                                                 307K   163K   283K   176K    139      0  12.5K   105K      1      0      0
2ms                                                55.7K   144K  59.2K   167K     63      0  12.1K   135K      6      0      0
4ms                                                75.3K   208K  93.5K   322K     96      0  15.4K   201K      7      0      0
8ms                                                 224K   527K   259K   978K    137      0  21.3K   316K      7      0      0
16ms                                                547K   936K   557K  1.06M    133      0  24.8K   363K     42      0      0
33ms                                                344K   931K   331K   657K    137      0  17.5K   348K     90      0      0
67ms                                               36.6K   653K  29.1K   350K     45      0  5.81K   247K    143      0      0
134ms                                              12.2K   241K  7.86K   122K     42      0  2.97K  64.9K    291      0      0
268ms                                              7.79K  68.4K  5.96K  22.0K      7      0    964  29.2K    661      0      0
536ms                                              5.40K  21.9K  3.88K  4.57K      7      0    227  11.1K  1.23K      0      0
1s                                                 3.55K  3.08K  1.16K  1.30K      0      0     44    595  2.34K      0      0
2s                                                 4.81K      6      0      0      0      0      0      0  4.82K      0      0
4s                                                 8.64K      0      0      0      0      0      0      0  8.63K      0      0
8s                                                 21.0K      0      0      0      0      0      0      0  21.0K      0      0
17s                                                31.8K      0      0      0      0      0      0      0  31.7K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

scsi-35000cca2a23c29a4                              total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  19.0K  26.4K   639K     47      0      0      0
511ns                                                  0      0      0      0   772K   203K  6.26M   429K     17      0      0
1us                                                    0      0      0      0   473K  9.92K  2.43M  1.07M     36      0      0
2us                                                    0      0      0      0   117K    417   166K   346K     12      0      0
4us                                                    0      0      0      0  2.87K  2.69K  8.04K  16.8K      1      0      0
8us                                                    0      0      0      0    347  1.06K  8.28K  5.52K      0      0      0
16us                                                   0      0      0      0    946     99  13.0K  9.21K      1      0      0
32us                                                   1      0      1      0  1.72K     39  24.3K  15.1K      0      0      0
65us                                               1.47M  8.31K  1.48M  8.69K     37      1  9.98K  20.5K      0      0      0
131us                                              2.53M  14.3K  2.51M  14.1K     34      0  22.6K  33.8K      1      0      0
262us                                              2.90M  31.5K  2.93M  37.4K     65      0  48.8K  47.9K      0      0      0
524us                                              2.63M   270K  2.65M   279K    127      0  29.2K  76.0K      1      0      0
1ms                                                 305K   173K   282K   186K    141      0  12.3K  99.8K      1      0      0
2ms                                                56.2K   143K  59.7K   165K    100      0  11.8K   131K      3      0      0
4ms                                                76.3K   205K  94.1K   312K    105      0  15.6K   194K      7      0      0
8ms                                                 222K   507K   257K   963K    130      0  21.4K   308K     16      0      0
16ms                                                531K   884K   542K  1.02M    157      0  24.5K   349K     31      0      0
33ms                                                360K   891K   346K   639K    106      0  17.9K   351K     73      0      0
67ms                                               39.3K   673K  31.4K   358K     55      0  6.52K   280K    163      0      0
134ms                                              12.9K   268K  7.78K   129K     77      0  3.30K  74.7K    323      0      0
268ms                                              7.76K  73.7K  5.98K  24.5K      9      0    863  30.0K    651      0      0
536ms                                              5.86K  23.6K  4.26K  4.64K      9      0    259  12.3K  1.29K      0      0
1s                                                 3.74K  3.71K  1.37K  1.53K      0      0     70    746  2.29K      0      0
2s                                                 4.66K      9      0      0      0      0      0      0  4.67K      0      0
4s                                                 9.00K      0      0      0      0      0      0      0  8.99K      0      0
8s                                                 21.3K      0      0      0      0      0      0      0  21.3K      0      0
17s                                                31.0K      0      0      0      0      0      0      0  31.0K      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------


mirror-2                                            total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  2.29K  75.5K    135   375K   192K      0      0
511ns                                                  0      0      0      0  21.7K   572K  3.82K  2.82M   892K      0      0
1us                                                    0      0      0      0  27.6K  76.8K  11.8K  2.32M   416K      0      0
2us                                                    0      0      0      0  5.43K  1.84K  1.11K   753K  18.6K      0      0
4us                                                    0      0      0      0    170  4.04K     89   384K  13.0K      0      0
8us                                                    0     28      0    426     35    915    105   688K  16.2K      0      0
16us                                                   2  1.95M      5  4.38M     75    126    215  1.23M  38.3K      0      0
32us                                                 100  4.63M    120  7.30M     95     41     89  1.58M  75.0K      0      0
65us                                               69.8K  3.81M  90.4K  2.85M     24      4     84  1.81M   132K      0      0
131us                                               448K  3.09M   498K  1.55M     24      0    100  1.95M   108K      0      0
262us                                               979K  1.89M  1005K   669K     51      0     92  1.14M  29.4K      0      0
524us                                               493K   653K   411K   502K     18      0     39   395K  2.21K      0      0
1ms                                                18.2K   195K  10.4K  16.6K      4      0     20   124K  1.04K      0      0
2ms                                                1.99K  66.4K    148  9.40K      0      0      3  47.5K  1.80K      0      0
4ms                                                2.54K  40.5K     72  2.64K      0      0      0  35.6K  2.38K      0      0
8ms                                                1.95K  58.3K      3  1.54K      0      0      0  56.4K  1.70K      0      0
16ms                                                 109   163K      0  1.76K      0      0      0   163K     90      0      0
33ms                                                   0   667K      0  1.73K      0      0      0   659K      0      0      0
67ms                                                   0  45.5K      0   1018      0      0      0  44.2K      0      0      0
134ms                                                  0  25.2K      0    474      0      0      0  24.7K      0      0      0
268ms                                                  0  16.1K      0     71      0      0      0  16.0K      0      0      0
536ms                                                  0  2.23K      0      0      0      0      0  2.23K      0      0      0
1s                                                     0      9      0      0      0      0      0      7      0      0      0
2s                                                     0      0      0      0      0      0      0      0      0      0      0
4s                                                     0      0      0      0      0      0      0      0      0      0      0
8s                                                     0      0      0      0      0      0      0      0      0      0      0
17s                                                    0      0      0      0      0      0      0      0      0      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

nvme-VO002000KWVVC_PHLJ227600232P0BGN_1             total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0    796  15.0K     50  82.8K  95.9K      0      0
511ns                                                  0      0      0      0  7.45K   170K  1.22K   605K   445K      0      0
1us                                                    0      0      0      0  9.43K  55.6K  3.90K  1.18M   209K      0      0
2us                                                    0      0      0      0  1.86K  1.57K    389   600K  9.20K      0      0
4us                                                    0      0      0      0     65  1.25K     33   159K  6.50K      0      0
8us                                                    0     12      0    213     10    356     31   295K  8.19K      0      0
16us                                                   0   821K      0  2.01M     21     46     73   495K  19.2K      0      0
32us                                                  49  1.88M     66  2.47M     32     13     26   610K  37.6K      0      0
65us                                               34.7K  1.55M  44.9K   973K     11      1     31   644K  66.1K      0      0
131us                                               219K  1023K   244K   211K      9      0     37   606K  54.1K      0      0
262us                                               484K   401K   497K  10.4K     13      0     28   224K  14.7K      0      0
524us                                               245K  18.8K   204K  4.67K      1      0     12  8.64K    987      0      0
1ms                                                9.05K  4.75K  5.36K  1.84K      0      0      2  2.67K    574      0      0
2ms                                                1.02K  2.65K     60    535      0      0      0  1.73K    963      0      0
4ms                                                1.26K  1.35K     30    302      0      0      0    766  1.15K      0      0
8ms                                                  947    529      1    476      0      0      0     96    851      0      0
16ms                                                 109    625      0    583      0      0      0     75     90      0      0
33ms                                                   0    643      0    596      0      0      0     53      0      0      0
67ms                                                   0    348      0    332      0      0      0     11      0      0      0
134ms                                                  0    156      0    146      0      0      0      0      0      0      0
268ms                                                  0     23      0     23      0      0      0      0      0      0      0
536ms                                                  0      0      0      0      0      0      0      0      0      0      0
1s                                                     0      0      0      0      0      0      0      0      0      0      0
2s                                                     0      0      0      0      0      0      0      0      0      0      0
4s                                                     0      0      0      0      0      0      0      0      0      0      0
8s                                                     0      0      0      0      0      0      0      0      0      0      0
17s                                                    0      0      0      0      0      0      0      0      0      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

nvme-VO002000KWVVC_PHLJ2276003G2P0BGN_1             total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0    783  30.2K     39   163K  96.0K      0      0
511ns                                                  0      0      0      0  7.26K   201K  1.27K  1.48M   447K      0      0
1us                                                    0      0      0      0  9.09K  10.8K  3.91K   749K   207K      0      0
2us                                                    0      0      0      0  1.74K    120    362   108K  9.38K      0      0
4us                                                    0      0      0      0     46  1.35K     24   146K  6.52K      0      0
8us                                                    0     16      0    213     11    302     36   250K  7.99K      0      0
16us                                                   2  1.14M      5  2.32M     29     31     71   455K  19.1K      0      0
32us                                                  40  1.67M     43  2.30M     32     12     37   641K  37.4K      0      0
65us                                               34.5K  1.42M  44.9K   857K      6      1     27   678K  65.8K      0      0
131us                                               219K  1.02M   244K   201K      9      0     37   620K  53.7K      0      0
262us                                               484K   398K   496K  8.59K     29      0     25   218K  14.7K      0      0
524us                                               245K  18.9K   204K  4.42K      4      0     10  8.98K  1.25K      0      0
1ms                                                8.80K  4.60K  4.71K  1.83K      0      0     10  2.56K    488      0      0
2ms                                                  956  2.39K     74    546      0      0      1  1.52K    879      0      0
4ms                                                1.28K  1.42K     31    295      0      0      0    800  1.23K      0      0
8ms                                                1.02K    508      1    444      0      0      0     92    888      0      0
16ms                                                   0    571      0    558      0      0      0     61      0      0      0
33ms                                                   0    614      0    592      0      0      0     38      0      0      0
67ms                                                   0    362      0    347      0      0      0      5      0      0      0
134ms                                                  0    168      0    160      0      0      0      0      0      0      0
268ms                                                  0     26      0     26      0      0      0      0      0      0      0
536ms                                                  0      0      0      0      0      0      0      0      0      0      0
1s                                                     0      0      0      0      0      0      0      0      0      0      0
2s                                                     0      0      0      0      0      0      0      0      0      0      0
4s                                                     0      0      0      0      0      0      0      0      0      0      0
8s                                                     0      0      0      0      0      0      0      0      0      0      0
17s                                                    0      0      0      0      0      0      0      0      0      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

nvme-INTEL_SSDPE2KX020T8_BTLJ73520B9F2P0BGN_1       total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0    763  30.3K     46   129K      0      0      0
511ns                                                  0      0      0      0  6.96K   201K  1.32K   767K      0      0      0
1us                                                    0      0      0      0  9.11K  10.3K  3.98K   419K      0      0      0
2us                                                    0      0      0      0  1.83K    149    384  45.3K      0      0      0
4us                                                    0      0      0      0     59  1.43K     32  78.7K      0      0      0
8us                                                    0      0      0      0     14    257     38   144K      0      0      0
16us                                                   0  15.1K      0  47.7K     25     49     71   312K      0      0      0
32us                                                  11  1.07M     11  2.54M     31     16     26   362K      0      0      0
65us                                                 599   854K    647  1.06M      7      2     26   531K      0      0      0
131us                                              9.37K  1.07M  9.45K  1.15M      6      0     26   776K      0      0      0
262us                                              11.9K  1.11M  11.8K   650K      9      0     39   729K      0      0      0
524us                                              2.60K   615K  2.60K   493K     13      0     17   378K      0      0      0
1ms                                                  337   186K    298  12.9K      4      0      8   118K      0      0      0
2ms                                                   29  61.3K     14  8.35K      0      0      2  44.3K      0      0      0
4ms                                                   11  37.7K     11  2.06K      0      0      0  34.1K      0      0      0
8ms                                                    1  57.3K      1    661      0      0      0  56.2K      0      0      0
16ms                                                   0   162K      0    662      0      0      0   163K      0      0      0
33ms                                                   0   665K      0    580      0      0      0   659K      0      0      0
67ms                                                   0  44.8K      0    339      0      0      0  44.2K      0      0      0
134ms                                                  0  24.9K      0    168      0      0      0  24.7K      0      0      0
268ms                                                  0  16.0K      0     22      0      0      0  16.0K      0      0      0
536ms                                                  0  2.23K      0      0      0      0      0  2.23K      0      0      0
1s                                                     0      9      0      0      0      0      0      7      0      0      0
2s                                                     0      0      0      0      0      0      0      0      0      0      0
4s                                                     0      0      0      0      0      0      0      0      0      0      0
8s                                                     0      0      0      0      0      0      0      0      0      0      0
17s                                                    0      0      0      0      0      0      0      0      0      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------


nvme-eui.5cd2e47fc4560100-part2                     total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0      6    233      0      0      0      0      0
511ns                                                  0      0      0      0      4  24.3K      0      1      0      0      0
1us                                                    0      0      0      0      4  68.6K      0      8      0      0      0
2us                                                    0      0      0      0      0  28.8K      0     13      0      0      0
4us                                                    0      0      0      0      0    449      0      1      0      0      0
8us                                                    0      0      0      0      0     49      0      1      0      0      0
16us                                                   0    135      0    302      0    671      0      0      0      0      0
32us                                                   1  24.4K      2  24.7K      0    556      0      0      0      0      0
65us                                                   4  22.4K      3  23.0K      0     46      0      3      0      0      0
131us                                                  2  42.4K      2  41.6K      0     81      0      7      0      0      0
262us                                                  1  31.2K      1  36.7K      0    158      0     22      0      0      0
524us                                                  2  1.75K      2  8.11K      0    263      0     31      0      0      0
1ms                                                    2  1.07K      2  1.01K      0    406      0     67      0      0      0
2ms                                                    2  1.30K      2  8.46K      0    651      0    125      0      0      0
4ms                                                    0  1.45K      0      0      0    967      0    262      0      0      0
8ms                                                    0  1.88K      0      0      0  1.12K      0    502      0      0      0
16ms                                                   0  2.35K      0      0      0  1.26K      0    998      0      0      0
33ms                                                   0  3.38K      0      0      0  1.30K      0  1.94K      0      0      0
67ms                                                   0  4.37K      0      0      0    727      0  3.64K      0      0      0
134ms                                                  0  4.71K      0      0      0    713      0  3.99K      0      0      0
268ms                                                  0  1.13K      0      0      0    443      0    691      0      0      0
536ms                                                  0      0      0      0      0      0      0      0      0      0      0
1s                                                     0      0      0      0      0      0      0      0      0      0      0
2s                                                     0      0      0      0      0      0      0      0      0      0      0
4s                                                     0      0      0      0      0      0      0      0      0      0      0
8s                                                     0      0      0      0      0      0      0      0      0      0      0
17s                                                    0      0      0      0      0      0      0      0      0      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------


nvme-VO002000KWVVC_PHLJ2276003G2P0BGN_5             total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  2.72K      1    236  9.30K     32      0      0
511ns                                                  0      0      0      0  7.29K      1  53.3K   337K  18.4K      0      0
1us                                                    0      0      0      0   111K      1   193K   517K  14.2K      0      0
2us                                                    0      0      0      0   153K      0   106K   358K    346      0      0
4us                                                    0      0      0      0  3.60K      0  4.07K  27.7K    106      0      0
8us                                                    0     11      0     20    100      0  2.30K  5.64K     93      0      0
16us                                                   0   129K      0   140K    192      0  1.59K  8.01K    303      0      0
32us                                                   0   139K      0   135K    577      0  2.28K  22.2K    490      0      0
65us                                               2.29K  60.9K  6.54K   165K     79      0  2.37K  30.1K    818      0      0
131us                                              30.2K   164K  46.4K  1.41M    188      0  4.64K  96.5K  1.15K      0      0
262us                                               156K   191K   181K   190K    315      0  7.68K   180K  1.46K      0      0
524us                                               214K   486K   231K  1011K    239      0  5.18K   338K  1.94K      0      0
1ms                                                 282K   900K   272K  1.18M     52      0  2.50K   536K  2.33K      0      0
2ms                                                24.1K   939K  19.3K   202K     16      0    349   852K  3.04K      0      0
4ms                                                4.62K   891K    851  15.8K     18      0     36   661K  3.57K      0      0
8ms                                                6.26K   590K     64    966      0      0     20   516K  6.11K      0      0
16ms                                               5.80K  14.0K      0      0      0      0      0  8.17K  5.49K      0      0
33ms                                               2.82K    110      0      0      0      0      0     92  2.83K      0      0
67ms                                               10.1K      0      0      0      0      0      0      0  10.1K      0      0
134ms                                              14.7K      0      0      0      0      0      0      0  14.6K      0      0
268ms                                              2.91K      0      0      0      0      0      0      0  2.91K      0      0
536ms                                                236      0      0      0      0      0      0      0    236      0      0
1s                                                    50      0      0      0      0      0      0      0     50      0      0
2s                                                     0      0      0      0      0      0      0      0      0      0      0
4s                                                     0      0      0      0      0      0      0      0      0      0      0
8s                                                     0      0      0      0      0      0      0      0      0      0      0
17s                                                    0      0      0      0      0      0      0      0      0      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

nvme-VO002000KWVVC_PHLJ227600232P0BGN_5             total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0  2.91K      2    252  9.61K     27      0      0
511ns                                                  0      0      0      0  6.66K      0  52.2K   325K  16.2K      0      0
1us                                                    0      0      0      0   105K      1   191K   521K  16.6K      0      0
2us                                                    0      0      0      0   158K      0   109K   365K    424      0      0
4us                                                    0      0      0      0  3.80K      0  4.18K  27.9K     99      0      0
8us                                                    0     14      0     24     93      0  2.31K  5.84K    123      0      0
16us                                                   0   129K      0   139K    201      0  1.62K  8.06K    352      0      0
32us                                                   0   138K      8   135K    580      0  2.32K  22.0K    560      0      0
65us                                               2.80K  62.6K  7.60K   165K     74      0  2.38K  30.3K    904      0      0
131us                                              30.6K   166K  46.2K  1.41M    183      0  4.59K  96.7K  1.18K      0      0
262us                                               156K   190K   181K   192K    380      0  7.70K   180K  1.55K      0      0
524us                                               214K   486K   231K  1007K    323      0  5.39K   339K  1.82K      0      0
1ms                                                 281K   900K   271K  1.18M    139      0  2.46K   537K  2.28K      0      0
2ms                                                24.0K   939K  19.3K   199K     32      0    343   853K  2.98K      0      0
4ms                                                4.49K   883K    820  15.2K      0      0     27   654K  3.50K      0      0
8ms                                                5.94K   598K     50  1.09K      0      0     19   523K  5.78K      0      0
16ms                                               5.32K  11.6K      0      6      0      0     19  6.60K  5.03K      0      0
33ms                                               3.20K    118      0      0      0      0      1    100  3.22K      0      0
67ms                                               10.0K      0      0      0      0      0      0      0  10.1K      0      0
134ms                                              15.8K      0      0      0      0      0      0      0  15.7K      0      0
268ms                                              2.94K      0      0      0      0      0      0      0  2.93K      0      0
536ms                                                172      0      0      0      0      0      0      0    172      0      0
1s                                                    80      0      0      0      0      0      0      0     80      0      0
2s                                                     0      0      0      0      0      0      0      0      0      0      0
4s                                                     0      0      0      0      0      0      0      0      0      0      0
8s                                                     0      0      0      0      0      0      0      0      0      0      0
17s                                                    0      0      0      0      0      0      0      0      0      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------

nvme-INTEL_SSDPE2KX020T8_BTLJ73520B9F2P0BGN_5       total_wait     disk_wait    syncq_wait    asyncq_wait
latency                                             read  write   read  write   read  write   read  write  scrub   trim  rebuild
-------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
1ns                                                    0      0      0      0      0      0      0      0      0      0      0
3ns                                                    0      0      0      0      0      0      0      0      0      0      0
7ns                                                    0      0      0      0      0      0      0      0      0      0      0
15ns                                                   0      0      0      0      0      0      0      0      0      0      0
31ns                                                   0      0      0      0      0      0      0      0      0      0      0
63ns                                                   0      0      0      0      0      0      0      0      0      0      0
127ns                                                  0      0      0      0      0      0      0      0      0      0      0
255ns                                                  0      0      0      0     84      1      5    663      0      0      0
511ns                                                  0      0      0      0    608      0  5.99K   190K      0      0      0
1us                                                    0      0      0      0  52.9K      2  90.9K   378K      0      0      0
2us                                                    0      0      0      0  99.2K      0  49.1K   369K      0      0      0
4us                                                    0      0      0      0  2.13K      0  1.69K  35.9K      0      0      0
8us                                                    0      0      0      0     70      0    166  3.68K      0      0      0
16us                                                   0      3      0    105    143      0    524  5.05K      0      0      0
32us                                                   0   173K      0   179K    169      0    669  8.23K      0      0      0
65us                                                 101  47.8K    112  45.0K      2      0     41  4.41K      0      0      0
131us                                              1.86K  32.7K  1.89K  31.6K      0      0     63  3.70K      0      0      0
262us                                              3.34K  9.87K  3.35K  20.3K      0      0    107  7.05K      0      0      0
524us                                              6.89K  26.3K  7.27K   441K      3      0    267  19.3K      0      0      0
1ms                                                1.16K  31.3K    961  27.3K      5      0    189  77.2K      0      0      0
2ms                                                 247K   308K   247K   419K     14      0    268   177K      0      0      0
4ms                                                43.4K   675K  44.2K  1.08M      7      0    505   307K      0      0      0
8ms                                                2.25K   549K  1.59K  23.6K      0      0     63   495K      0      0      0
16ms                                                 111   443K     16  15.7K      0      0     44   217K      0      0      0
33ms                                                  84  8.29K      0      0      0      0     83  4.90K      0      0      0
67ms                                                  29  1.36K      0      0      0      0     12  1.24K      0      0      0
134ms                                                  0      0      0      0      0      0      0      0      0      0      0
268ms                                                  0      0      0      0      0      0      0      0      0      0      0
536ms                                                  0      0      0      0      0      0      0      0      0      0      0
1s                                                     0      0      0      0      0      0      0      0      0      0      0
2s                                                     0      0      0      0      0      0      0      0      0      0      0
4s                                                     0      0      0      0      0      0      0      0      0      0      0
8s                                                     0      0      0      0      0      0      0      0      0      0      0
17s                                                    0      0      0      0      0      0      0      0      0      0      0
34s                                                    0      0      0      0      0      0      0      0      0      0      0
68s                                                    0      0      0      0      0      0      0      0      0      0      0
137s                                                   0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------------------------
pve#

ekke · Feb 27, 2024

gea said:
- If you remove a special vdev, data is moved to the pool.
- A vdev lost is otherwise a pool lost as data is not cached there but stored.
- a 3way mirror is not perfect for read performance but allow 2 disks to fail.

Gea, any feedback on this:

pve# zpool get cachefile
NAME PROPERTY VALUE SOURCE
rpool cachefile - default
safe cachefile - default
speed cachefile - default
tank cachefile none local
temp cachefile - default

none cachefile?! not default? hm odd.

ekke · Feb 27, 2024

I turned off all lxc and VMs. rebooted.

did more testing and read speeds are good now, 100MBps for each drive in zpool iostat -v

pve# fio --name=seqread --rw=read --direct=1 --ioengine=libaio --bs=8k --numjobs=1 --size=30G --runtime=120 --group_reporting
seqread: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
seqread: Laying out IO file (1 file / 30720MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=1200MiB/s][r=154k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=25199: Wed Feb 28 00:40:20 2024
read: IOPS=120k, BW=940MiB/s (985MB/s)(30.0GiB/32687msec)
slat (usec): min=2, max=89855, avg= 7.11, stdev=130.04
clat (nsec): min=658, max=96097, avg=929.54, stdev=1437.12
lat (usec): min=3, max=89862, avg= 8.04, stdev=130.08
clat percentiles (nsec):
| 1.00th=[ 684], 5.00th=[ 700], 10.00th=[ 716], 20.00th=[ 732],
| 30.00th=[ 756], 40.00th=[ 772], 50.00th=[ 796], 60.00th=[ 820],
| 70.00th=[ 836], 80.00th=[ 860], 90.00th=[ 876], 95.00th=[ 908],
| 99.00th=[ 2544], 99.50th=[16320], 99.90th=[20096], 99.95th=[22144],
| 99.99th=[27520]
bw ( KiB/s): min=613296, max=1424720, per=99.95%, avg=961865.85, stdev=97906.08, samples=65
iops : min=76662, max=178090, avg=120233.35, stdev=12238.12, samples=65
lat (nsec) : 750=27.95%, 1000=69.39%
lat (usec) : 2=1.59%, 4=0.13%, 10=0.30%, 20=0.53%, 50=0.10%
lat (usec) : 100=0.01%
cpu : usr=17.31%, sys=76.93%, ctx=542, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3932160,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=940MiB/s (985MB/s), 940MiB/s-940MiB/s (985MB/s-985MB/s), io=30.0GiB (32.2GB), run=32687-32687msec
pve#

thulle · Feb 27, 2024

ekke said:
These are all my zfetch values:

pve# cat /sys/module/zfs/parameters/zfetch*
67108864
67108864

I mean you can't expect that much of a change if you're not changing the value - just inserting the default value?

Another way to boost read speed can be to increase zfs_vdev_async_read_max_active. But doing that at the cost of write latency, which may give complex secondary effects.

Search

optimize zfs for seq reads

ekke

Member

ekke

Member

ekke

Member

BackupProphet

Well-Known Member

SnJ9MX

Active Member

gea

Well-Known Member

thulle

Member

ekke

Member

ekke

Member

ekke

Member

ekke

Member

ekke

Member

ekke

Member

thulle

Member