optimize zfs for seq reads

[ekke]

Im looking at zfs tunables to increase the readahead. Already got 1M record size etc.

reads currently at almost 300MBps. , writes > 1 GBps

need to increase the readaheadcache, and perhapds other paramters.

echo 67108864 > /sys/module/zfs/parameters/zfetch_max_distance ?

 

[ekke]

ok just tried with
echo 67108864 > /sys/module/zfs/parameters/zfetch_max_distance
didnt increase the seq reads.

 

[ekke]

Added

options zfs zfs_arc_max=103079215104
vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_write_max=26214400
vfs.zfs.l2arc_write_boost=52428800

need to boot for it to get used though.

Now I just need to increase the readahead abit, cant find which tunable to adjust for that

 

[BackupProphet]

The defaults should be really good, and is what I use to saturate 100G link, with 160 hard drives.

 

[ekke]

well Its not. as I stated I can get almost the same read as writes, so Im missing about 600MBps

reads maxout at 300MBps (each drive at 30 35 MBps ) writes at 1100 MBps (each drive at almsot 120MBps)


Im looking at zfetch_min_distance currently that should be increased I think.

 

[BackupProphet]

Do you use l2arc, do you have special metadata devices?

 

[ekke]

Yes both special and l2arc, should not be that relevant, but now I'm looking to increase the per disk drive seq read speed, currently at 30 MBps

Get it closer to the per disk write speed of 150MBps.

The 10tb hgst sas drives can peak at 200 .

 

[BackupProphet]

L2arc can slow down performance. It's a very tricky feature and should always be benchmarked. Anyway, I think the main reason you have slow read performance is one or more of your hard drives is not working right. Does zpool events tell you something?

 

[gea]

ZFS Arc or L2Arc defaults for read ahead are quite optimal for all sort of workloads. You can modify for a certain workload but this can decrease performance on other workloads. In general these settings can increase performance for a few percent only.

If you say that your read performance is 30 MB/s then this is bad. If you can tune to increase 30% then you are at 40 MB/s what is bad as well. I do not asume that 30% is possible.

I would first look why this is so low. Maybe one or some disks are bad. Check iostat on load if all disks have same load. You can also build a pool per disk and compare performance. When ZFS writes to a pool it spreads data over the pool ao achieve a constant performance with many files and users. ZFS does not optimize for a single user with a large file. The datasheet disk values are lab values when you fill a disk track ba track. This is not the case with Raid or ZFS. But in general you should achieve at least 80 MB/s per disk.

Mostly sequential or large fileread performance depend on pool layout like n-way mirrors where ZFS can read from all mirror disks concurrently (a mirror can have 2x single disk performance)) or n-vdev Raid-Z and raw single single disk performance. A 1M recsice can help a little over a default 128K with the price of inefficiency on small files.

Arc and L2Arc tunings can help a little on small io but cache is for most/last read ZFS datablocks not files. Mostly performance is bad for small io. L2Arc can help a little as it is persistent or on low RAM situations. If you have enough RAM or not a large number of users with volatile files ex a university mailserver, forget L2Arc. A special vdev mirror is a better option. If you set a small blocksize for example of 64GB with a recsize of 128K+, then all files with size up to 64K (nearly all "office files") are stored on the special vdev, not only metadata. If you set 64K and a recsize 64K, all files of this filesystem are on the special vdev so you can set per filesystem workload.

But first, investigate why average single disk performance is so low.

 

[BackupProphet]

Another thing that can slow down performance is having very large block sizes. In my benchmarks, even for sequential reads, having the recordsize set to 128KB is optimal compared to 1MB+, going for 4MB or even 16MB can significantln slow down sequential read performance. For video I think setting the recordsize to 1MB is fine, but for vectorized databases, 128KB was like 2x faster than 1MB.

 

[ekke]

L2arc can slow down performance. It's a very tricky feature and should always be benchmarked. Anyway, I think the main reason you have slow read performance is one or more of your hard drives is not working right. Does zpool events tell you something?

Feb 18 2024 20:04:27.144891454 ereport.fs.zfs.zpool
Feb 18 2024 20:04:36.344772850 ereport.fs.zfs.zpool
Feb 18 2024 20:04:46.620640371 ereport.fs.zfs.zpool
Feb 18 2024 20:04:56.864508308 ereport.fs.zfs.zpool
Feb 18 2024 20:05:07.108376243 ereport.fs.zfs.zpool
Feb 18 2024 20:05:16.328257381 ereport.fs.zfs.zpool
Feb 18 2024 20:05:26.608124852 ereport.fs.zfs.zpool
Feb 18 2024 20:05:36.799993458 ereport.fs.zfs.zpool
Feb 18 2024 20:05:47.079860931 ereport.fs.zfs.zpool
Feb 18 2024 20:05:56.311741913 ereport.fs.zfs.zpool
Feb 18 2024 20:06:06.519610314 ereport.fs.zfs.zpool
Feb 18 2024 20:06:16.743478508 ereport.fs.zfs.zpool
Feb 18 2024 20:06:26.991346392 ereport.fs.zfs.zpool
Feb 18 2024 20:06:36.231227422 ereport.fs.zfs.zpool
Feb 18 2024 20:06:46.495095663 ereport.fs.zfs.zpool
Feb 18 2024 20:06:56.722964360 ereport.fs.zfs.zpool
Feb 18 2024 20:07:06.954833010 ereport.fs.zfs.zpool
Feb 18 2024 20:07:17.186701662 ereport.fs.zfs.zpool
Feb 18 2024 20:07:26.402583357 ereport.fs.zfs.zpool
Feb 18 2024 20:07:36.634452007 ereport.fs.zfs.zpool
Feb 18 2024 20:07:46.894320296 ereport.fs.zfs.zpool
Feb 18 2024 20:07:57.150188639 ereport.fs.zfs.zpool
Feb 18 2024 20:08:06.390070027 ereport.fs.zfs.zpool
Feb 18 2024 20:08:16.601938936 ereport.fs.zfs.zpool
Feb 18 2024 20:08:26.825807687 ereport.fs.zfs.zpool
Feb 18 2024 20:08:37.065676237 ereport.fs.zfs.zpool
Feb 18 2024 20:08:46.341557161 ereport.fs.zfs.zpool
Feb 18 2024 20:08:56.565425914 ereport.fs.zfs.zpool
Feb 18 2024 20:09:06.821294256 ereport.fs.zfs.zpool
Feb 18 2024 20:09:17.033163165 ereport.fs.zfs.zpool
Feb 18 2024 20:09:26.261044705 ereport.fs.zfs.zpool
Feb 18 2024 20:09:36.500913250 ereport.fs.zfs.zpool
Feb 18 2024 20:09:46.740781799 ereport.fs.zfs.zpool
Feb 18 2024 20:09:56.984650296 ereport.fs.zfs.zpool
Feb 18 2024 20:10:06.252531321 ereport.fs.zfs.zpool
Feb 18 2024 20:10:16.492399867 ereport.fs.zfs.zpool
Feb 18 2024 20:10:26.700268827 ereport.fs.zfs.zpool
Feb 18 2024 20:10:36.924137583 ereport.fs.zfs.zpool
Feb 18 2024 20:10:47.168006081 ereport.fs.zfs.zpool
Feb 18 2024 20:10:56.399887574 ereport.fs.zfs.zpool
Feb 18 2024 20:11:06.655755916 ereport.fs.zfs.zpool
Feb 18 2024 20:11:16.871624772 ereport.fs.zfs.zpool
Feb 18 2024 20:11:27.127493119 ereport.fs.zfs.zpool
Feb 18 2024 20:11:36.343374814 ereport.fs.zfs.zpool
Feb 18 2024 20:11:46.531244031 ereport.fs.zfs.zpool
Feb 18 2024 20:11:56.795112270 ereport.fs.zfs.zpool
Feb 18 2024 20:12:07.054980565 ereport.fs.zfs.zpool
Feb 18 2024 20:12:16.290862001 ereport.fs.zfs.zpool
Feb 18 2024 20:12:26.566730087 ereport.fs.zfs.zpool
Feb 18 2024 20:12:36.766599150 ereport.fs.zfs.zpool
Feb 18 2024 20:12:47.010467646 ereport.fs.zfs.zpool
Feb 18 2024 20:12:56.258348931 ereport.fs.zfs.zpool
Feb 18 2024 20:13:06.502217427 ereport.fs.zfs.zpool
Feb 18 2024 20:13:16.750085872 ereport.fs.zfs.zpool
Feb 18 2024 20:13:26.969954678 ereport.fs.zfs.zpool
Feb 18 2024 20:13:36.205836114 ereport.fs.zfs.zpool
Feb 18 2024 20:13:46.445704661 ereport.fs.zfs.zpool
Feb 18 2024 20:13:56.645573722 ereport.fs.zfs.zpool
Feb 18 2024 20:14:06.877442375 ereport.fs.zfs.zpool
Feb 18 2024 20:14:17.133310718 ereport.fs.zfs.zpool
Feb 18 2024 20:14:26.381192003 ereport.fs.zfs.zpool
Feb 18 2024 20:14:36.593060909 ereport.fs.zfs.zpool
Feb 18 2024 20:14:46.828929511 ereport.fs.zfs.zpool
Feb 18 2024 20:14:57.088797804 ereport.fs.zfs.zpool
Feb 18 2024 20:15:06.284679753 ereport.fs.zfs.zpool
Feb 18 2024 20:15:16.520549008 ereport.fs.zfs.zpool
Feb 18 2024 20:15:26.740421836 ereport.fs.zfs.zpool
Feb 18 2024 20:15:36.984294367 ereport.fs.zfs.zpool
Feb 18 2024 20:15:46.256178988 ereport.fs.zfs.zpool
Feb 18 2024 20:15:56.464051961 ereport.fs.zfs.zpool
Feb 18 2024 20:16:06.707924491 ereport.fs.zfs.zpool
Feb 18 2024 20:16:16.987796573 ereport.fs.zfs.zpool
Feb 18 2024 20:16:26.219681695 ereport.fs.zfs.zpool
Feb 18 2024 20:16:36.467554172 ereport.fs.zfs.zpool
Feb 18 2024 20:16:46.687426998 ereport.fs.zfs.zpool
Feb 18 2024 20:16:56.915299724 ereport.fs.zfs.zpool
Feb 18 2024 20:17:07.127172650 ereport.fs.zfs.zpool
Feb 18 2024 20:17:16.383057470 ereport.fs.zfs.zpool
Feb 18 2024 20:17:26.614930150 ereport.fs.zfs.zpool
Feb 18 2024 20:17:36.842802878 ereport.fs.zfs.zpool
Feb 18 2024 20:17:47.086675409 ereport.fs.zfs.zpool
Feb 18 2024 20:17:56.282560980 ereport.fs.zfs.zpool
Feb 18 2024 20:18:06.526433506 ereport.fs.zfs.zpool
Feb 18 2024 20:18:16.766306082 ereport.fs.zfs.zpool
Feb 18 2024 20:18:27.026178410 ereport.fs.zfs.zpool
Feb 18 2024 20:18:36.222063979 ereport.fs.zfs.zpool
Feb 18 2024 20:18:46.469936462 ereport.fs.zfs.zpool
Feb 18 2024 20:18:56.725808842 ereport.fs.zfs.zpool
Feb 18 2024 20:19:06.965681420 ereport.fs.zfs.zpool
Feb 18 2024 20:19:17.185554246 ereport.fs.zfs.zpool
Feb 18 2024 20:19:26.409439468 ereport.fs.zfs.zpool
Feb 18 2024 20:19:36.681311648 ereport.fs.zfs.zpool
Feb 18 2024 20:19:46.921184224 ereport.fs.zfs.zpool
Feb 18 2024 20:19:57.113057401 ereport.fs.zfs.zpool
Feb 18 2024 20:20:06.380940564 ereport.fs.zfs.zpool
Feb 18 2024 20:20:16.592809604 ereport.fs.zfs.zpool
Feb 18 2024 20:20:26.824678389 ereport.fs.zfs.zpool
Feb 18 2024 20:20:37.044547330 ereport.fs.zfs.zpool
Feb 18 2024 20:20:46.272428988 ereport.fs.zfs.zpool
Feb 18 2024 20:20:56.524297516 ereport.fs.zfs.zpool
Feb 18 2024 20:21:06.780165990 ereport.fs.zfs.zpool
Feb 18 2024 20:21:17.008034828 ereport.fs.zfs.zpool
Feb 18 2024 20:21:26.247916335 ereport.fs.zfs.zpool
Feb 18 2024 20:21:36.479785121 ereport.fs.zfs.zpool
Feb 18 2024 20:21:46.731653648 ereport.fs.zfs.zpool
Feb 18 2024 20:21:56.959522487 ereport.fs.zfs.zpool
Feb 18 2024 20:22:07.175391478 ereport.fs.zfs.zpool
Feb 18 2024 20:22:16.375273499 ereport.fs.zfs.zpool
Feb 18 2024 20:22:26.659141623 ereport.fs.zfs.zpool
Feb 18 2024 20:22:36.891010406 ereport.fs.zfs.zpool
Feb 18 2024 20:22:47.102879447 ereport.fs.zfs.zpool
Feb 18 2024 20:22:56.350760848 ereport.fs.zfs.zpool
Feb 18 2024 20:23:06.622629119 ereport.fs.zfs.zpool
Feb 18 2024 20:23:16.846498008 ereport.fs.zfs.zpool
Feb 18 2024 20:23:27.034367357 ereport.fs.zfs.zpool
Feb 18 2024 20:23:36.310248402 ereport.fs.zfs.zpool
Feb 18 2024 20:23:46.546117138 ereport.fs.zfs.zpool
Feb 18 2024 20:23:56.761986206 ereport.fs.zfs.zpool
Feb 18 2024 20:24:07.065854155 ereport.fs.zfs.zpool
Feb 18 2024 20:24:16.273736150 ereport.fs.zfs.zpool
Feb 18 2024 20:24:26.501605072 ereport.fs.zfs.zpool
Feb 18 2024 20:24:36.761473586 ereport.fs.zfs.zpool
Feb 18 2024 20:24:47.001342355 ereport.fs.zfs.zpool
Feb 18 2024 20:24:56.229224092 ereport.fs.zfs.zpool
Feb 18 2024 20:25:06.505092398 ereport.fs.zfs.zpool
Feb 18 2024 20:25:16.708961628 ereport.fs.zfs.zpool
Feb 18 2024 20:25:26.964830190 ereport.fs.zfs.zpool
Feb 18 2024 20:25:36.172712183 ereport.fs.zfs.zpool
Feb 18 2024 20:25:46.404581054 ereport.fs.zfs.zpool
Feb 18 2024 20:25:56.664449567 ereport.fs.zfs.zpool
Feb 18 2024 20:26:06.920318130 ereport.fs.zfs.zpool
Feb 18 2024 20:26:17.140187155 ereport.fs.zfs.zpool
Feb 18 2024 20:26:26.404068430 ereport.fs.zfs.zpool
Feb 18 2024 20:26:36.603937710 ereport.fs.zfs.zpool
Feb 18 2024 20:26:46.859806275 ereport.fs.zfs.zpool
Feb 18 2024 20:26:57.083675246 ereport.fs.zfs.zpool
Feb 18 2024 20:27:06.327556778 ereport.fs.zfs.zpool
Feb 18 2024 20:27:16.563425598 ereport.fs.zfs.zpool
Feb 18 2024 20:27:26.811294272 ereport.fs.zfs.zpool
Feb 18 2024 20:27:37.031163292 ereport.fs.zfs.zpool
Feb 18 2024 20:27:46.295044568 ereport.fs.zfs.zpool
Feb 18 2024 20:27:56.494913849 ereport.fs.zfs.zpool
Feb 18 2024 20:28:06.738782565 ereport.fs.zfs.zpool
Feb 18 2024 20:28:16.954651640 ereport.fs.zfs.zpool
Feb 18 2024 20:28:26.198533174 ereport.fs.zfs.zpool
Feb 18 2024 20:28:36.442401890 ereport.fs.zfs.zpool
Feb 18 2024 20:28:46.706270353 ereport.fs.zfs.zpool
Feb 18 2024 20:28:56.958138967 ereport.fs.zfs.zpool
Feb 18 2024 20:29:07.190007838 ereport.fs.zfs.zpool
Feb 18 2024 20:29:09.317980567 sysevent.fs.zfs.config_sync
Feb 18 2024 20:29:09.329980413 sysevent.fs.zfs.config_sync
Feb 18 2024 20:29:09.405979439 sysevent.fs.zfs.vdev_attach
Feb 18 2024 20:29:09.413979336 sysevent.fs.zfs.resilver_start
Feb 18 2024 20:29:09.413979336 sysevent.fs.zfs.history_event
Feb 18 2024 20:29:09.489978363 sysevent.fs.zfs.config_sync
Feb 18 2024 20:29:10.041971288 sysevent.fs.zfs.history_event
Feb 18 2024 20:29:10.041971288 sysevent.fs.zfs.history_event
Feb 18 2024 20:29:10.041971288 sysevent.fs.zfs.resilver_finish
Feb 18 2024 20:29:10.049971186 sysevent.fs.zfs.vdev_remove
Feb 18 2024 20:29:10.061971032 sysevent.fs.zfs.config_sync
Feb 18 2024 20:29:15.177905465 sysevent.fs.zfs.history_event
Feb 18 2024 20:29:16.445889215 ereport.fs.zfs.zpool
Feb 18 2024 20:29:26.653758392 ereport.fs.zfs.zpool
Feb 18 2024 20:29:36.917626852 ereport.fs.zfs.zpool
Feb 18 2024 20:29:47.113496186 ereport.fs.zfs.zpool
Feb 18 2024 20:29:56.405377106 ereport.fs.zfs.zpool
Feb 18 2024 20:30:06.633246031 ereport.fs.zfs.zpool
Feb 18 2024 20:30:16.869114854 ereport.fs.zfs.zpool
Feb 18 2024 20:30:27.140983212 ereport.fs.zfs.zpool
Feb 18 2024 20:30:36.356865102 ereport.fs.zfs.zpool
Feb 18 2024 20:30:47.864717621 ereport.fs.zfs.zpool
Feb 18 2024 20:30:57.104599205 ereport.fs.zfs.zpool
Feb 18 2024 20:31:06.356480635 ereport.fs.zfs.zpool
Feb 18 2024 20:31:16.600349353 ereport.fs.zfs.zpool
Feb 18 2024 20:31:26.804218585 ereport.fs.zfs.zpool
Feb 18 2024 20:31:37.036087457 ereport.fs.zfs.zpool
Feb 18 2024 20:31:46.259969248 ereport.fs.zfs.zpool
Feb 18 2024 20:31:56.483838217 ereport.fs.zfs.zpool
Feb 18 2024 20:32:06.739706780 ereport.fs.zfs.zpool
Feb 18 2024 20:32:16.967575700 ereport.fs.zfs.zpool
Feb 18 2024 20:32:26.171457757 ereport.fs.zfs.zpool
Feb 18 2024 20:32:36.411326581 ereport.fs.zfs.zpool
Feb 18 2024 20:32:46.651195404 ereport.fs.zfs.zpool
Feb 18 2024 20:32:56.875064430 ereport.fs.zfs.zpool
Feb 18 2024 20:33:07.146932840 ereport.fs.zfs.zpool
Feb 18 2024 20:33:16.350814935 ereport.fs.zfs.zpool
Feb 18 2024 20:33:26.602683605 ereport.fs.zfs.zpool
Feb 18 2024 20:33:36.846552380 ereport.fs.zfs.zpool
Feb 18 2024 20:33:47.066421459 ereport.fs.zfs.zpool
Feb 18 2024 20:33:56.298303194 ereport.fs.zfs.zpool
Feb 18 2024 20:34:06.526172167 ereport.fs.zfs.zpool
Feb 18 2024 20:34:16.754041146 ereport.fs.zfs.zpool
Feb 18 2024 20:34:27.049909252 ereport.fs.zfs.zpool
Feb 18 2024 20:34:36.281790988 ereport.fs.zfs.zpool
Feb 18 2024 20:34:46.529659710 ereport.fs.zfs.zpool
Feb 18 2024 20:34:56.737528943 ereport.fs.zfs.zpool
Feb 18 2024 20:35:07.001397461 ereport.fs.zfs.zpool
Feb 18 2024 20:35:16.229279249 ereport.fs.zfs.zpool
Feb 18 2024 20:35:26.489147817 ereport.fs.zfs.zpool
Feb 18 2024 20:35:36.697017048 ereport.fs.zfs.zpool
Feb 18 2024 20:35:46.920886077 ereport.fs.zfs.zpool
Feb 18 2024 20:35:57.156754948 ereport.fs.zfs.zpool
Feb 18 2024 20:36:06.404636479 ereport.fs.zfs.zpool
Feb 18 2024 20:36:16.668504991 ereport.fs.zfs.zpool
Feb 18 2024 20:36:26.908373817 ereport.fs.zfs.zpool
Feb 18 2024 20:36:37.100243253 ereport.fs.zfs.zpool
Feb 18 2024 20:36:46.368124528 ereport.fs.zfs.zpool
Feb 18 2024 20:36:56.615993252 ereport.fs.zfs.zpool
Feb 18 2024 20:37:06.875861818 ereport.fs.zfs.zpool
Feb 18 2024 20:37:17.135730390 ereport.fs.zfs.zpool
Feb 18 2024 20:37:26.351612329 ereport.fs.zfs.zpool
Feb 18 2024 20:37:36.571481411 ereport.fs.zfs.zpool
Feb 18 2024 20:37:46.791350494 ereport.fs.zfs.zpool
Feb 18 2024 20:37:57.075218756 ereport.fs.zfs.zpool
Feb 18 2024 20:38:06.319100337 ereport.fs.zfs.zpool
Feb 18 2024 20:38:16.558969158 ereport.fs.zfs.zpool
Feb 18 2024 20:38:26.806837878 ereport.fs.zfs.zpool
Feb 18 2024 20:38:37.082706243 ereport.fs.zfs.zpool
Feb 18 2024 20:38:46.326587823 ereport.fs.zfs.zpool
Feb 18 2024 20:38:56.534457061 ereport.fs.zfs.zpool
Feb 18 2024 20:39:06.774325884 ereport.fs.zfs.zpool
Feb 18 2024 20:39:17.022194604 ereport.fs.zfs.zpool
Feb 18 2024 20:39:26.266076187 ereport.fs.zfs.zpool
Feb 18 2024 20:39:36.505945012 ereport.fs.zfs.zpool
Feb 18 2024 20:39:46.721814145 ereport.fs.zfs.zpool
Feb 18 2024 20:39:56.969682864 ereport.fs.zfs.zpool
Feb 18 2024 20:40:06.233564191 ereport.fs.zfs.zpool
Feb 18 2024 20:40:16.465433116 ereport.fs.zfs.zpool
Feb 18 2024 20:40:26.689302142 ereport.fs.zfs.zpool
Feb 18 2024 20:40:36.945170761 ereport.fs.zfs.zpool
Feb 18 2024 20:40:46.181052445 ereport.fs.zfs.zpool
Feb 18 2024 20:40:56.404921473 ereport.fs.zfs.zpool
Feb 18 2024 20:41:06.632790429 ereport.fs.zfs.zpool
Feb 18 2024 20:41:16.880659111 ereport.fs.zfs.zpool
Feb 18 2024 20:41:27.088528306 ereport.fs.zfs.zpool
Feb 18 2024 20:41:36.308410160 ereport.fs.zfs.zpool
Feb 18 2024 20:41:46.556278840 ereport.fs.zfs.zpool
Feb 18 2024 20:41:56.824147262 ereport.fs.zfs.zpool
Feb 18 2024 20:42:07.016016660 ereport.fs.zfs.zpool
Feb 18 2024 20:42:16.271898052 ereport.fs.zfs.zpool
Feb 18 2024 20:42:26.475767294 ereport.fs.zfs.zpool
Feb 18 2024 20:42:36.747635666 ereport.fs.zfs.zpool
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.pool_create
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.763571389 sysevent.fs.zfs.history_event
Feb 18 2024 20:42:41.771571287 sysevent.fs.zfs.config_sync
Feb 18 2024 20:42:46.951504911 ereport.fs.zfs.zpool
Feb 18 2024 20:42:56.203386354 ereport.fs.zfs.zpool
Feb 18 2024 20:43:06.411255550 ereport.fs.zfs.zpool
Feb 18 2024 20:43:16.675124028 ereport.fs.zfs.zpool
Feb 18 2024 20:43:26.922992711 ereport.fs.zfs.zpool
Feb 18 2024 20:43:37.134861856 ereport.fs.zfs.zpool
Feb 18 2024 20:43:46.374743456 ereport.fs.zfs.zpool
Feb 18 2024 20:43:56.654611726 ereport.fs.zfs.zpool
Feb 18 2024 20:44:06.878480716 ereport.fs.zfs.zpool
Feb 18 2024 20:44:17.114349550 ereport.fs.zfs.zpool
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.pool_create
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.022286659 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.026286608 sysevent.fs.zfs.history_event
Feb 18 2024 20:44:22.034286506 sysevent.fs.zfs.config_sync
Feb 18 2024 21:08:37.607603994 sysevent.fs.zfs.history_event
Feb 18 2024 21:08:46.287491962 sysevent.fs.zfs.history_event
Feb 18 2024 21:09:32.946889754 sysevent.fs.zfs.history_event
Feb 18 2024 21:09:43.194757491 sysevent.fs.zfs.history_event
Feb 18 2024 21:11:53.085081072 sysevent.fs.zfs.history_event
Feb 18 2024 21:12:03.848942150 sysevent.fs.zfs.history_event
Feb 18 2024 21:35:22.574876581 sysevent.fs.zfs.history_event
Feb 18 2024 21:35:56.046442102 sysevent.fs.zfs.history_event
Feb 18 2024 23:03:21.147402371 sysevent.fs.zfs.history_event
Feb 18 2024 23:04:02.358872495 sysevent.fs.zfs.history_event
Feb 20 2024 21:45:48.737723112 sysevent.fs.zfs.history_event
Feb 21 2024 12:31:19.221031667 sysevent.fs.zfs.history_event
Feb 21 2024 12:31:19.229031564 sysevent.fs.zfs.history_event
Feb 21 2024 12:31:19.237031463 sysevent.fs.zfs.history_event
Feb 21 2024 12:31:19.237031463 sysevent.fs.zfs.history_event
Feb 21 2024 12:34:20.662711514 sysevent.fs.zfs.history_event
Feb 21 2024 12:34:20.670711412 sysevent.fs.zfs.history_event
Feb 21 2024 12:34:20.670711412 sysevent.fs.zfs.history_event
Feb 21 2024 12:34:20.670711412 sysevent.fs.zfs.history_event
Feb 21 2024 13:03:56.728003381 sysevent.fs.zfs.history_event
Feb 21 2024 13:05:05.495123414 sysevent.fs.zfs.history_event
Feb 21 2024 13:15:33.019096881 sysevent.fs.zfs.history_event
Feb 21 2024 13:15:33.219094324 sysevent.fs.zfs.history_event
Feb 21 2024 13:58:50.222014557 sysevent.fs.zfs.history_event
Feb 21 2024 13:58:50.226014507 sysevent.fs.zfs.history_event
Feb 21 2024 13:58:50.230014456 sysevent.fs.zfs.history_event
Feb 21 2024 13:58:50.230014456 sysevent.fs.zfs.history_event
Feb 21 2024 14:08:25.934741393 sysevent.fs.zfs.history_event
Feb 21 2024 14:08:25.942741291 sysevent.fs.zfs.history_event
Feb 21 2024 14:08:25.946741241 sysevent.fs.zfs.history_event
Feb 21 2024 14:08:25.946741241 sysevent.fs.zfs.history_event
Feb 21 2024 15:08:51.216778016 ereport.fs.zfs.io
Feb 21 2024 18:40:55.930689982 sysevent.fs.zfs.history_event
Feb 21 2024 18:40:55.934689928 sysevent.fs.zfs.history_event
Feb 21 2024 18:40:55.938689873 sysevent.fs.zfs.history_event
Feb 21 2024 18:40:55.938689873 sysevent.fs.zfs.history_event
Feb 21 2024 19:28:02.239844943 sysevent.fs.zfs.history_event
Feb 21 2024 19:28:19.215610409 sysevent.fs.zfs.history_event
Feb 21 2024 19:29:40.206492456 sysevent.fs.zfs.history_event
Feb 21 2024 19:30:01.114204487 sysevent.fs.zfs.history_event
Feb 21 2024 19:34:01.410894746 sysevent.fs.zfs.history_event
Feb 21 2024 19:34:15.486700875 sysevent.fs.zfs.history_event
Feb 21 2024 21:47:29.104935225 sysevent.fs.zfs.history_event
Feb 21 2024 22:10:36.109901773 sysevent.fs.zfs.history_event
Feb 21 2024 22:11:32.657120859 sysevent.fs.zfs.history_event
Feb 21 2024 22:13:05.991831636 sysevent.fs.zfs.history_event
Feb 21 2024 22:13:32.979458445 sysevent.fs.zfs.history_event
Feb 21 2024 22:50:27.260951028 sysevent.fs.zfs.config_sync
Feb 21 2024 22:50:27.260951028 sysevent.fs.zfs.vdev_add
Feb 22 2024 21:13:05.885279837 sysevent.fs.zfs.history_event
Feb 22 2024 21:13:10.509224586 sysevent.fs.zfs.history_event
Feb 22 2024 21:42:39.128100146 sysevent.fs.zfs.config_sync
Feb 24 2024 11:45:50.454986717 ereport.fs.zfs.io
Feb 24 2024 14:28:03.676079443 sysevent.fs.zfs.history_event
Feb 24 2024 14:56:13.620075848 sysevent.fs.zfs.config_sync
Feb 24 2024 14:58:52.698243639 sysevent.fs.zfs.history_event
Feb 24 2024 14:58:59.986159699 sysevent.fs.zfs.history_event
Feb 24 2024 14:59:27.413843796 sysevent.fs.zfs.history_event
Feb 24 2024 14:59:34.425763035 sysevent.fs.zfs.history_event
Feb 24 2024 14:59:43.249661405 sysevent.fs.zfs.history_event
Feb 24 2024 15:00:52.996858080 sysevent.fs.zfs.history_event
Feb 24 2024 15:00:55.380830623 sysevent.fs.zfs.history_event
Feb 25 2024 10:51:33.791620561 sysevent.fs.zfs.history_event
Feb 25 2024 10:52:10.899173930 sysevent.fs.zfs.history_event
Feb 25 2024 10:52:39.534829150 sysevent.fs.zfs.history_event
Feb 25 2024 10:53:01.214566762 sysevent.fs.zfs.history_event
Feb 25 2024 10:53:57.445886199 sysevent.fs.zfs.history_event
Feb 25 2024 10:54:27.273525196 sysevent.fs.zfs.history_event
Feb 25 2024 10:55:08.349028061 sysevent.fs.zfs.history_event
Feb 25 2024 10:56:22.404131777 sysevent.fs.zfs.history_event

# zpool status -x
all pools are healthy

all the drives reads at the same speed 30-35MBps


I think soon my "workload" will have a moving 300GB of warm data, hot data will be in arc.
so I think soon l2arc will be fine for my usecase.

 

[ekke]

Another thing that can slow down performance is having very large block sizes. In my benchmarks, even for sequential reads, having the recordsize set to 128KB is optimal compared to 1MB+, going for 4MB or even 16MB can significantln slow down sequential read performance. For video I think setting the recordsize to 1MB is fine, but for vectorized databases, 128KB was like 2x faster than 1MB.

I have large files only here. 5-100GB reading these files should be faster, not as fast as writes, but still faster.

 

[ekke]

these ereports seems to be more related to my ssds.

ie

eb 24 2024 11:45:50.454986717 ereport.fs.zfs.io
        class = "ereport.fs.zfs.io"
        ena = 0x36c310e32b204801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x6f6563eda2199348
                vdev = 0x75933d9b3d515596
        (end detector)
        pool = "tank"
        pool_guid = 0x6f6563eda2199348
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x75933d9b3d515596
        vdev_type = "disk"
        vdev_path = "/dev/disk/by-id/nvme-INTEL_SSDPE2KX020T8_BTLJ73520B9F2P0BGN_1-part1"
        vdev_devid = "nvme-INTEL_SSDPE2KX020T8_BTLJ73520B9F2P0BGN-part1"
        vdev_ashift = 0xc
        vdev_complete_ts = 0x1c36c310d87aa
        vdev_delta_ts = 0xf9aeb
        vdev_read_errors = 0x0
        vdev_write_errors = 0x0
        vdev_cksum_errors = 0x0
        vdev_delays = 0x0
        parent_guid = 0x8fad4512dd0bc9d
        parent_type = "mirror"
        vdev_spare_paths =
        vdev_spare_guids =
        zio_err = 0x3d
        zio_flags = 0x180180
        zio_stage = 0x2000000
        zio_pipeline = 0x2100000
        zio_delay = 0x299c6
        zio_timestamp = 0x1c36c30f6bea6
        zio_delta = 0x29b4a
        zio_priority = 0x0
        zio_offset = 0x5d638000
        zio_size = 0x1000
        zio_objset = 0x709
        zio_object = 0x0
        zio_level = 0x0
        zio_blkid = 0x1204
        time = 0x65d9c8de 0x1b1e8bdd
        eid = 0x4bf

Interesting, these u.2 drives are attached via pci converter.

wait Blocks all I/O access until the device connectivity is recovered and the errors are cleared with zpool clear. This is the default behavior.

 

[ekke]

and they are old, one 24th and one 21th so nothing when doing these tests.

 

[ekke]

the resilver was due to a nvme drive failure

 

[gb00s]

Tbh, this zpool setup is the weirdest and wrongest looking I've ever seen.

Special ... 3* so no full functioning mirror. Ooopsi
Log ... 1* no mirror. Ooopsi
Cache ... 3* hopefully in Raid0 and not Z1

What do the tests say without special, log and cache?

 

[BackupProphet]

He probably mirrors 3 special meta devices as he doesnt know that ZFS store an additional copy of the metadata by default. Or that he knows about it, and has set redundant_metadata=none

Log doesnt need mirror.

Cache is supposed to be invidual drives.

 

[ekke]

Tbh, this zpool setup is the weirdest and wrongest looking I've ever seen.

Special ... 3* so no full functioning mirror. Ooopsi
Log ... 1* no mirror. Ooopsi
Cache ... 3* hopefully in Raid0 and not Z1

What do the tests say without special, log and cache?

3way mirror for special vdev.

slog doesn't need mirror in my usecase. optane highly unlikely to die, very seldom sync writes.

cache drives for the rest of data on the drives.

all these drives are namspaces from 3 2TB u.2 drives.


special drives cant be removed, they are vdev, vdevs cant be removed w/o pool failing.

 

[gea]

3way mirror for special vdev.

that is the perfect solution regarding security and read performance

slog doesn't need mirror in my usecase. optane highly unlikely to die, very seldom sync writes.

korrekt. Only in a very special case of a crash with an slog failure, commited writes are lost.

cache drives for the rest of data on the drives.

if you cannot add more RAM as RAM cache is faster than Disk cache and Diskcache (L2Arc) needs RAM to operate.

special drives cant be removed, they are vdev, vdevs cant be removed w/o pool failing.

You can remove but only when ashift is the same on all vdevs

 

[gb00s]

That special devices can't be removed is so wrong as the earth is a disc. Special devs in mirror and same ashift can be removed. I had much better performance and sec with specials devs in Raid10 than a x-way mirror w/Optane 4800x and DCPM (don't ask me why and how).

Edit:

What if you get rid of all the extra devs, specially the special devs?