Prevent or reduce frequency of txg_sync

[MelBourbon]

I have a small server (Debian Buster) containing 2 WD RED hard disk which I have encrypted with LUKS and on which I set up a ZFS datapool with RAID 1.

As the server is not used to frequently I have created a autoshutdown script which checks for services etc. and if the data drives are active or in standby. If data drives are in standby for 40 minutes the server shuts down. This worked until recently where the drives are not staying in standby longer than 5 minutes.

With `iotop` I figured out that this is related to a `txg_sync` command by ZFS which wakes up my drives. Can I reduce the frequency of this command running to allow my drives to go to standby again? Like I said it worked before and I don't know why not now anymore.

 

[Stephan]

Instead of iotop try "fatrace -c" after chdir'ing into your ZFS mount point. This will show which process is still accessing your disks, preventing disk sleep. txg_sync is just the symptom, something wants to write data and ZFS syncs it to disk using this and other threads.

To debug further also stop all processes accessing the mount point, unmount ZFS and then observe if disks are going to sleep at all. If you can't get them to sleep automatically after a while, try hd-idle rewrite in Go.

 

[MelBourbon]

Instead of iotop try "fatrace -c" after chdir'ing into your ZFS mount point. This will show which process is still accessing your disks, preventing disk sleep. txg_sync is just the symptom, something wants to write data and ZFS syncs it to disk using this and other threads.

Thanks. Yes, I checked already that the drives are going in standby with hdparm -y. Now I did a try with fatrace. My datapool which is based on the drives is mounted on /media/disk1 and has several datasets. I'm running in one terminal a script which shows the status of the drives, in one terminal iotop and in the third I went into /media/disk1 and run fatrace -c. Then I send the drives in standby. Unfortunately after a few minutes the drives went active again, txg_sync was showing up on iotop but I did not get any output with fatrace. Do I need to change dir into the datasets and check them individually?

 

[Stephan]

Yes, try changing into the datasets. There is also a chance that fatrace does not work as good with ZFS as with ext4 say. I run a RAIDZ2 and also put drives into standby (no permanent I/O is going on except nightly borgbackup), all on SATA ports and using hd-idle. Works problem-free and as expected. Arch Linux on LTS kernel 5.4.85 but I don't think that is relevant here. Drives wake up and are sent back to sleep when no I/O is going on. In other words there is no weird I/O by ZFS itself when nothing is going on that would keep disks away. You probably have some process still accessing the dataset. Got some snapshotting going on? I would say stop all suspicious daemons and if that fixes it, start them one by one until problem returns.

 

[MelBourbon]

Ok, thanks. Yes, I also want to figure out which daemon is causing this. It's not only about shutting down the server. I have also the problem that my WD red are going in standby independent (even without hdparm or hd-idle). I would like to avoid that they are waked up and start running so often if the server is idle even if the server is running.

Thanks for your support. I was afraid doing such a detailed debugging (killing processes and daemons) to figure this out as it always takes up to 15 minutes to confirm. But now it seems I don't have any option. Because using fatrace -c in the datasets also did not showed anything.

I don't have any snapshotting ongoing.

Edit: Seems to be a known issue with fatrace and ZFS (see Support for fanotify · Issue #6079 · openzfs/zfs). So I will need to do more checks.

Edit 2: By using `iosnoop` (brendangregg/perf-tools) I was able to identify that dmcrypt is writing to my disk in regular intervals. Please note that my underlying disks are encrypted with LUKS. Need to investigate how to resolve.

./iosnoop -d 8,16
Tracing block I/O. Ctrl-C to end.
COMM         PID    TYPE DEV      BLOCK        BYTES     LATms
dmcrypt_writ 1895   W    8,16     2080476248   4096    6516.10
dmcrypt_writ 1895   W    8,16     3334728264   4096    6516.14
dmcrypt_writ 1895   W    8,16     2080429048   16384      0.16
dmcrypt_writ 1895   W    8,16     3334728272   20480      0.21
dmcrypt_writ 1895   W    8,16     2080476256   20480      0.16
dmcrypt_writ 1895   W    8,16     3328225336   16384      0.20