Napp-it 4ai/client server ZFS web-gui for any OS or mixed cluster

[gea]

Not everyone has a cluster with a dozen servers. Regardless of the OS, whether it’s a 1 GB mini IoT device or a petabyte system, whether using Storage Spaces, S3 object storage, or ZFS—not everyone wants to manage everything centrally with simple, fast, encrypted file sync or any-to-any ZFS replication (e.g., via GitHub - guenther-alka/cs-stream: Encrypted TCP stream transport for ZFS replication (replaces netcat)).

However, that is precisely our target audience with napp-it cs. S3 integration, in particular, is a hot topic right now. It allows you to publish local files to the internet easily and securely (e.g., setting up a private cloud with RustFS). RustFS can also run independently in a "set-and-forget" mode to keep two servers or buckets in bidirectional sync. High availability and real-time backups over the internet or LAN, plain and simple.

What’s currently missing is the bridge between "shared multi-user SMB with ACLs" and object storage, which inherently lacks multi-user locking with file ACLs. This is where I'm stepping in with the cs-sync module.

With cs-sync, my goal is to keep two folders—such as an SMB share on a NAS and an S3 share—in sync in real time using event triggers (reacting instantly on change, without needing full file comparisons), including SMB ACLs. Regardless of the OS, this is now possible thanks to Go, with the option for real-time automated backups of changes over the network to a backup system (see GitHub - guenther-alka/cs-sync: Realtime bidirectional folder sync for ZFS hosts (napp-it CS)).

I would love to get your feedback or suggestions on this concept! A first preview featuring napp-it cs and real-time sync is coming soon. Anyone interested in serious testing should consider getting Claude Pro along with the Filesystem and Desktop Commander extensions/agents (on Windows or other OSs via Plink/SCP SSH access for Claude to inspect scripts, logs, and members) for analysis, stress testing, audits, etc.

 

[gea]

3 object storage features eg RustFS (current 1.0 rc11 beta) offers

- Internet Access for files on a ZFS fileserver
- webconsole for remote file access
- backup target with encryption, snaps and dedup (any OS backup/restore)
- set and forget Site/Bucket bidirectional replication between RustFS hosts

S3 lacks:
- multiuser access with file locking and ACL

So the conclusion is:
To use SMB/ZFS and S3 with same data on either service, you need a bidir sync service
that mirrors data and at least folder ACL with inheritance between S3 and SMB shares

This is what I included in napp-it cs 26.06 07.29 rc
Update: Menu About > Frontend Update

-missing private or osx menus
-osx setup: https://www.napp-it.org/pdf/apple_osx_en.pdf

Go based multi os cs-stream job

Releases · guenther-alka/cs-stream

Encrypted TCP stream transport for ZFS replication (replaces netcat) - guenther-alka/cs-stream

github.com

-ZFS replication or filebased sync (via rclone)
-encrypted, buffered or io limited tunnel any to any

Go based multi os cs-sync background service

Releases · guenther-alka/cs-sync

Realtime bidirectional folder sync for ZFS hosts (napp-it CS) - guenther-alka/cs-sync

github.com

-event driven local uni/bi service with folder ACL support for realtime sync (eg SMB and S3 shares)
-event driven remote uni sync service with folder ACL support for realtime backup (encrypted)

Status ample (current and last 10m/h/day/week)
Pool, Cap, Disk, IO, RAM, Jobs, Sync, Last

Main menu "Servergroup" shows running sync services on members

Bidir realtime sync (optionally with an additional realtime backup primary filesystem > backupserver)

 

[gea]

The new event driven (on file modifications) realtime cs-sync in napp-it cs is not only a perfect method to keep local S3 object storage and local SMB shares bidirectionally in sync but a new approach for realtime sync between data and backup, best paired with ZFS snaps on destination to have backup versions eg snaps hourly-keep 24, daily-keep 24, ..

remains an important aspect:

Behavior on slow or unstable connections

Handling disconnects and slow links is a core design goal, not an afterthought -- local-only 1.x never had to deal with this at all. See cs-sync-2.0-design.info section 3 for the full rationale.

• Disconnects (flaky WLAN, link flaps) are normal operation, not errors. No timeout, no alarm, just a log line. Every change that can't reach the remote right now goes into a persisted pending queue that survives both process restart and host reboot.
• Coalesced per path. A file that changes 100 times while offline is transferred once, latest state only -- the queue stores "this path needs sync," not an event log.
• Reconnect uses exponential backoff (1s -> 2s -> ... capped at 5min) before a retry counts against a file's failure count, so a flapping link can't burn through the retry budget in seconds.
• Backpressure, not blocking. The filesystem watcher never blocks on the network -- it can produce changes faster than a slow link can carry them indefinitely; the sender just drains the queue at whatever speed the link allows.
• Queue depth is unbounded by design. A deep queue just means "slow link + lots of data," which has to keep working. The retry limit (10 attempts, see backoff above) applies only to individual files that keep failing for their own reasons (permission errors, unreadable source, etc.) -- such a file is quarantined and logged so it stops blocking the rest of the queue, everything else keeps syncing.
• Atomic writes + end-to-end hash. Every transfer goes to a temp file, is hash-verified, then renamed into place -- a drop mid-transfer never leaves a half-written file that size/mtime could later mistake for current.
• Torn-copy detection. If the source changes while a slow transfer is still reading it, a before/after size+mtime mismatch discards the copy and re-queues it -- no locking needed.
• --bwlimit throttles a remote leg so an initial full sync over a narrow/shared link doesn't saturate it for everything else using the same connection.

Not yet implemented: delta transfer for large partially-changed files (a changed file is always sent in full -- see Known gaps above) and a queue disk-space warning threshold for very long outages. (an option for a next release for very large files)

 

[gea]

Coming from ESXi, there are two things I'm used to / that matter to me:

1. When I take a ZFS snap of a VM storage, it shouldn't just contain the virtual disks but also the VM configuration, so I can restore a VM from a ZFS snap (creating a snap, unlike PBS, is instant with nothing to copy, and even zfs send over LAN is much faster). On ESXi that was a .vmx file in the VM folder. On Proxmox, VM configurations live in /etc/pve.

Solution: I've added a snap option "include regular folder" to the snap job. This lets me capture the state of /etc/pve at snap time along with the snap.

Another use case would be Windows, to have c:/Users/"me" with personal documents on NTFS included in the ZFS snap.

1. Consistent VM snaps
   A ZFS snap is always consistent for the host managing the ZFS pool, thanks to copy-on-write and ZFS's internal snapshot handling. It looks completely different for VMs and their guest filesystems on ZFS. A ZFS snapshot on the host doesn't care whether their filesystems stay consistent. They end up in the snap in the same state as after an unexpected power outage. The risk of corrupted data can be reduced by making sure no write operations happen during the snap (VM freeze). Proxmox has various methods for that. I've now integrated zfs snapshot into a Go-based tool that automatically provides various freeze options to reduce the likelihood of a corrupted VM in the snap. The only way to avoid it entirely would be to snap while offline, or to also save RAM state — but with current RAM sizes that's very slow.

Go tool (included in current napp-it CS in autosnap and replication jobs)
GitHub - guenther-alka/cs-freeze4snap: Consistent ZFS snapshots for Proxmox VM/LXC guests - freeze if possible, snapshot regardless

 

[kapone]

@gea - Once again, my friend, you're trying to do things in your product that can be solved...well...in other ways. Like Ansible.

Proxmox (it is a major target for ZFS based systems, please don't get me started on the hot mess that's Truenas...) just...doesn't have enough to be comparable to vSphere/ESX. Long way off. As far as Solaris/OpenIndiana/OmniOS/whatever are concerned, they're dead and buried.

So. And So. That's where we stand. Anybody trying to play with Proxmox at scale isn't trying to use napp-it (or anything like it). They simply build their automation pipeline with tools that can do it (and there's many, choose your poison).

I'm a prime example of what I just said. I run Proxmox...at scale...is debatable, if two sites, each with ~28 nodes and 3 different clusters don't count. I had to build my own tooling, because Proxmox...well, just doesn't have it. It's a good product, but way off from the likes of ESXi.

 

[gea]

There's more than one way to skin a cat.

 

[kapone]

There's more than one way to skin a cat.

And?

Edit: p.s. VMs are fairly dead. Containers are where it's at as we speak. So, this may be a moot point.

 

[gea]

The only real VM you need in Proxmox could be a storage vm eg TN, OmniOS or Windows (napp-it cs is there to avoid this) or a Windows client VM.

 

[gea]

napp-it cs_rc 26.08.12
- cs-sync can sync paths and path with remote cs-sync or remote RustFS bucket
- cs-send, report status and alert over tls mail and/or chat
(Discord, Slack,ntfy,gotify,Telegram)

GitHub - guenther-alka/cs-send: Send mail (SMTP/TLS, HTML+attachments) and chat alerts (Discord/Telegram/Slack/ntfy/Gotify) from the command line
universal cs-tools for any OS, part of napp-it cs but can be used outside

 

[gea]

napp-it cs has now all major features of napp-it se (Solaris edition since 2009)
plus many new unique features, update see menu About > Update

napp-it cs 26.08.23.15
- Localisation en,cn,de,es,fr,it,ru
starting with menus (About > Settings
(with en switch in topmenu)

26.08.21.23
-security audit and hardening
-iSCSI targets (Linux,Illumos,Solaris)
-iSCSI initiators (Linux,Illumos,Solaris,Windows)
-Target zvol -> replicate -> import target=preserve target LU/NAA

 

[gea]

napp-it cs 26.08.27.01
-cs-tools no longer included, download via About > Download CS Tools
(cs-aihelp + ollama lokal ai, cs-freez4snap, cs-send, cs-sleeper, cs-stream, cs-sync)
cs-aihelp (AI Helpdesk):
- fullscreen menu Help > AI Helpdesk with plan and act mode (AI can run local commands on members)
- read only widget on any page
- two selectable providers including local Ollama

Setup AI Helpdesk

GitHub - guenther-alka/cs-aihelp: AI Helpdesk module for napp-it cs: chat helpdesk in the web-GUI with free/cloud LLM providers, RAG, optional web research and chat history.

AI Helpdesk module for napp-it cs: chat helpdesk in the web-GUI with free/cloud LLM providers, RAG, optional web research and chat history. - guenther-alka/cs-aihelp

github.com

- download cs-aihelp + Ollama optionally (lokal AI) in menu About > Download CS Tools
- configure System > Service > AI Helpdesk



I have added quick AI analyse buttons for pool, disk, OS, CS, job and member stat for selected member

 

[gea]

napp-it_4ai_rc_26.09.06.09
- cs-console for any member OS (first rc)
GitHub - guenther-alka/cs-console: Ephemeral, per-request interactive PTY relay backend for napp-it cs (server.pl get_tty)

Console and Expect mode for interactive commands eg change user pw

Update in About > Frontend Update (update dev release)
Use menu About > Download cs tools to download cs console on member
Open Menu System > Console for a full intractive console on member

napp-it_4ai_rc_26.09.03.20
- media indexer-job, robust resume reindex on bad indexed files from last run
(file error, vision error)
- AI indexer (Go app) update to 0.4
improved load handling (avoud 100% load for indexer on gpu less hardware)
- media selector, select from found criteria in media

napp-it_4ai_rc_26.09.02.13 dev release only
pre-release (quite a lot of new features)

- name changed from "napp-it cs" to "napp-it 4ai (client server web-gui)"

Use Cases
In-house storage server or cluster (any / mixed OS)
In-house S3 cloud server or cluster (any / mixed OS)
In-house AI provider or cluster (any / mixed OS)


napp-it 4ai add on tools
- cs-proxy for https webgui and inhouse OpenAI provider, always included
(faster than Perl SSL and really copy and run)

all other cs tools as download (About > Download cs tools)

- cs-aihelp (AI Helpdesk widget or chatpage with file/terminalaccess)
any OS Go daemon -- AI core, /ask, Level 2 exec proposer.

- cs-sleeper (disk spin-down)
any OS Go daemon, Disk spin-down / wake-up daemon (System > Power)

- cs-sync (realtime sync)
any OS Go app, Realtime sync service (System > Services > Realtime Sync).

- cs-send (mail / chat otify)
any OS Go app Notification / mail helper.

- cs-stream (event stream)
any OS Go app, Event stream service.

- cs-freeze4snap
any OS Go app, Filesystem freeze for snapshots (Proxmox only).

- cs-imageindex (photo/video indexer))
Photo/video folder indexer: EXIF/GPS, vision-LLM scene description, face recognition.

- cs-scripts
e.g Llama local KI installer for OmniOS
GitHub - guenther-alka/cs-scripts


remote cs-console (any/mixed OS)

 

[gea]

napp-it 4ai (client/server)

Downloads — napp-it

The new CS tools—especially the AI Helpdesk with optional AI console access as root/administrator upon confirmation and manual password entry (after all, the AI shouldn't know everything, making it ideal for troubleshooting and setup)—as well as a true interactive console (like PuTTY for each of the 8 supported OS options)—are now included in the standard napp-it 4ai (client/server).

CS tools (independent binaries, also run outside of napp-it)
https://www.napp-it.org/cs-tools.html

 

[gea]

Update cs-aihelpdesk auf 1.2.7
improved ui and ai communication

https://github.com/guenther-alka/cs-aihelp

CS-Tools & Scripts — napp-it

Update:
About>Update Frontend
About>Update cs-tools

 

[gea]

Napp-it 4ai (client-server) supports any OS for the frontend web GUI and backend remote management, including mixed clusters ranging from FreeBSD, Linux, Illumos/OmniOS, macOS, and Solaris to Windows Desktop or Server—optimized for Arm64 or X64 where appropriate. However, for newcomers or major system adjustments, the same question always arises: which OS, and partly, which hardware and file system?

When you have a choice—either because it is a custom-configured system where the OS and NAS are not tightly coupled (like Synology, Qnap, etc.) or because you can install any OS you like—this is how I see it:

Hardware

• Architecture: X86 preferred over Arm (except for Arm on macOS).
• RAM: 4–16 GB for a pure NAS; 16–128 GB for All-in-One (Hypervisor + NAS, e.g., with ESXi or Proxmox); 48–196 GB for an in-house AI provider (Arm Mac mini/Studio); ECC RAM if budget and hardware permit.
• Form Factor: Tiny hardware (e.g., Raspberry Pi) is possible but heavily limiting; uATX/ATX mainboards with AMD/Intel CPUs are much better.
• Drive Bays: Hot-swap drive bays are very practical.
• Drive Configuration: 1x OS drive (64 GB+) + 2x NVMe (special vdev) + 2 HDDs (data) or more.

File System

• ZFS: Clearly the superior choice. In terms of features and stability, it outperforms NTFS, ext3, XFS, and newer file systems like Btrfs or ReFS.
• S3 Integration: A sensible addition is S3 for in-house internet clouds backed by ZFS on the LAN.

Mainstream OS Options

• Linux (for the latest ZFS features): My favorite distribution is Proxmox because a great hypervisor and ZFS are always included out of the box.
• Windows 11 Pro or Workstation (Desktop): Great because everyone knows it, though ZFS is still at the release candidate stage; test thoroughly for your intended use case beforehand.
• Windows Server (affordable Essentials): Ideal if you need SMB Direct and 25/100G networking for 4K/8K video editing.

Specialist OS Options

• OmniOS: Particularly robust, minimalist, and serves as an exceptional SMB server with top-tier ACL support (a Solaris fork with native SMB, NFS, and ZFS).
• FreeBSD: If you know and love it; still slightly faster and more robust than Linux.
• macOS: If you know and love it, or if you also need an in-house AI provider.

 

[gea]

Disclaimer: This is not AI generated text but my own suggestion, AI is partly trained with my experience eg my 3M Thread https://hardforum.com/threads/openz...-storage-spaces-with-napp-it-web-gui.1573272/

ask Gemini : Is Gea a well known ZFS and Storage specialist?

Gemini:
"Yes, Gea is very well known in the storage and ZFS community.

He is best known as the creator and developer of napp-it, a premier web-based management GUI and appliance tool for ZFS (and enterprise storage). For well over a decade, he has been a prominent figure on storage forums—particularly on ServeTheHome and ComputerBase—where he is widely recognized as an expert voice on ZFS implementations, cross-platform file sharing (SMB/NFS), and storage management across operating systems like Illumos/OmniOS, Solaris, Proxmox/Linux, FreeBSD, and Windows."

 

[gea]

Howto

1. Jobs with napp-it 4ai (Client-Server)
Download Link (jobs & services.pdf)

Jobs are created in the Jobs menu and can be filtered and displayed by job type. They are stored and executed on the frontend. Jobs can be run manually, time-based via the auto.pl service, or at boot time. Parameters can be modified by clicking on the job ID.

The Jobs menu provides an overview of recent job actions:

• Clicking on a job type (e.g., replicate) displays the recent logs for that specific job.
• Clicking on "last" shows the details of the most recent job execution.

Backup jobs typically follow the 3-2-1 rule as a minimum requirement: three copies of the data (ideally with snapshot versioning: hourly for today, daily for this month, etc.), stored on two different storage media, with one of them kept off-site/externally.

Cross-Platform Support via cs-tools
Napp-it 4ai client-server supports backup and restore jobs on mixed servers running any operating system—ranging from FreeBSD, Linux, Illumos, and macOS to Solaris and Windows. Uniform OS procedures are usually difficult or impossible here because every OS provides different tools, variants, and options.

This challenge is solved using cs-tools for each OS, which not only provide a uniform interface but also capabilities that are otherwise unavailable out-of-the-box.

• Napp-it backup jobs process options using cs-tools (single-file binaries built with Go for each OS).
• cs-tools are an integrated part of napp-it 4ai/cs, but they can also be used externally (e.g., in custom scripts or batch files).
• cs-tools Documentation

2. Services
Services are background services that are started either on demand or automatically along with napp-it.

(More to come)

2.1 ZFS Replication
The "traditional" method of data transfer in ZFS (Menu: Jobs > Replication)

In this process, two ZFS datasets (filesystems or zvols) are kept in sync locally or across the network. It relies on ZFS snapshots and transfers only new or modified ZFS data blocks as a data stream, making it exceptionally fast. Even petabyte-scale high-load servers containing millions of files can be kept in sync within minutes, including open files in their latest state.

Over the network, administrators often use unencrypted mbuffer or netcat, or encrypted ssh, for data transport. In principle, OpenZFS can replicate with Illumos ZFS and Qnap ZFS as independent ZFS developments, but not with original Oracle Solaris ZFS.

• cs-stream Tool: Replaces standard utilities across all operating systems. It handles netcat/mbuffer (faster buffered transfer), ssh (encryption), or pv (throttled data rates to prevent a slow internet connection from being blocked). Unencrypted ZFS filesystems are also securely transmitted via a one-time password-encrypted tunnel. For more details, see cs-stream on GitHub.

Replication Settings (hold and keep)
When creating a replication job (any server to any server), you can configure times, transport settings, as well as keep and hold rules per job.

• Replication snapshots are numbered sequentially per job.
• The last two snapshots are always retained.
• To continue an incremental replication, an identical snapshot must exist on both sides (matching job ID and matching sequence number).
• hold: Defines the number of the newest replication snapshots to be retained, where $n$ represents days (e.g., 8) or $ns$ represents snapshot count (e.g., 8s for the last 8 snapshots).
• keep: Retention rules based on hours, days, months, or years (e.g., hours:24,days:32,months:12,years:2):
  • hours:24: One snapshot is protected for each of the last 24 hours.
  • days:32: One snapshot is protected for each of the last 32 days.
  • months:12: One snapshot is protected for each of the last 12 months.
  • years:2: One snapshot is protected for each of the last 2 years.

Duplicate snapshots are not retained if multiple rules apply to the same snapshot. Older replication snapshots that are protected by neither a hold nor a keep rule are deleted.

The blocking keep rule is embedded in the snapshot name. For example:

daten1/backup_iso/nvme_nfs@1716449451_repli_zfs-kd0830_omnios_nr_439

This represents a replication of the job with ID 1716449451 and sequence number 439 for August 30th of the current year.

2.2 Filesync Jobs
Based on file comparison

Filesync can be executed at any time and has no prerequisites, such as requiring an identical base snapshot like replication does.

• Filesync is a dedicated job type that also runs on top of an encrypted cs-stream tunnel and can utilize snapshots.
• ACLs are supported between identical source and destination operating systems by saving and restoring them via a .csv file.
• Filesync operates at high speeds by utilizing parallel streams.
• rclone (rsync for clouds) is used as the underlying sync tool
  
  
  
  
  
  
  
  
  
  

 

[gea]

Remote Backup via the Internet

With a good backup, at least one copy of the data is remote—meaning not on the same power grid, fire compartment, or building, and ideally not even on the same LAN. ZFS replication or file sync are ideal within a LAN, but over the internet, encryption and a firewall or a secure tunnel should generally be used. While VPN or cs-stream + firewall can also enable secure and encrypted ZFS replication or file sync, an S3-based approach is ideal for the internet because:

• Fixed ports are used for data transfer, e.g., for RustFS, port 9000 for file transfer and 9001 for the web console.
• Every file is an encrypted, authenticated HTTPS URL, making it perfect for web-based cloud usage.
• S3 works well with poor, slow network connections or Wi-Fi.

To use S3, you need an S3 server. This can be Amazon AWS (expensive), an affordable German S3 internet provider, or an in-house/home S3 server like MinIO or RustFS. Napp-it supports one or more RustFS servers in single-node/multi-node cluster mode as well as single/multiple disk mode (ideal: ZFS pool = RustFS single-disk mode).

There are three options for backups or data transfers via S3:

1. WinSCP (Quick and Easy)
Use an S3 server in WinSCP, ideally via SSL/TLS so that transfers are encrypted, and configure the S3 path mode in the advanced settings. WinSCP then operates in a two-pane mode comparing local folders vs. S3 buckets.



2. File Sync with rclone
Rclone is like rsync for the cloud. It can synchronize local folders and S3 buckets. Data transfer is encrypted using HTTPS. It features simple handling, and the data in the S3 bucket can be read with any S3 tools—making it ideal if you want to access it from anywhere. Napp-it can create S3 backup jobs that synchronize a local folder with an S3 bucket using rclone.

3. Backup with restic
Restic is a backup tool that creates an encrypted data repository in a bucket on the S3 server. Inside this repository, data is fundamentally encrypted with the client's password after deduplication and stored with snapshot versioning. Snapshots + encryption + deduplication are restic's key features, making it an ideal backup alternative to ZFS methods. When creating a napp-it S3 backup/restore job, you can choose restic in addition to rclone, and select a snapshot for restores. Restic backup takes a bit longer on the first run due to deduplication, but from the second run onwards, it is very fast and space-saving because only modified data blocks are processed.

Bucket / Site Replication

With two S3 RustFS servers, bucket or site replication provides a real-time high-availability option. You can configure RustFS so that two buckets or two entire servers (all buckets) are kept in sync independently and event-based in a two-way mode—truly a "set and forget" solution.

 

[gea]

4ai/cs_rc_26.10.08.14 (update: menu About > Frontend Update, or cli base installer)

smaller Bugfixes, readonly S3 to RustFS buckets/snaps or regular folder/snaps
Manual rework

Details
4ai/cs_rc_26.10.08.14

-new manual napp-it: https://www.napp-it.org/pdf/napp-it-cs_en.pdf
-new manual cs-team v.0.61: https://www.napp-it.org/pdf/cs-team_en.pdf

-bugfix in menu About > Settings (missing descriptions)
-cache invalidate prior and after zpool or zfs actions to guarantee correct gui state
-cs-s3gateway, ro access to zfs snaps of RustFS buckets or regular folders

 

[gea]

Question to Claude AI:
Why is my Proxmox autosnap job 1780659397 throwing an error?

I could check the logs now—the local job log in the Windows frontend, or perhaps the logs on the Proxmox member—and would probably figure out the cause even without AI. However, there are more exciting activities than analyzing log files and troubleshooting complex problems.

Response from Claude within 10 seconds: Job 1780659397 is a daily snapshot job on the rpool of member proxmox_1 (192.168.2.112). It runs Sundays at 11:00 PM with keep=2, hold=8, VM-Freeze=yes, and the additional option include=/etc/pve. It is currently failing, not because of the settings, but due to an outdated program version on the member.

I read the job files and the log.

Findings

• Error: Today's manual run at 10:45 AM failed with flag provided but not defined: -pre-snap-cmd. Because of include=/etc/pve, the job passes the --pre-snap-cmd option to cs-freeze4snap so that /etc/pve is included in the snapshot via rsync. The cs-freeze4snap tool on the Proxmox member does not recognize this option because it is older than version 1.3.0. Your local Windows program is version 1.3.0 and recognizes it.
• Last Success: The last successful run was on 08/30/2026 (rpool@i_daily-…), and prior to that on 06/05.
• Missing Auto-Runs: For the Sundays of 09/20, 09/27, and 10/04, only the auto-request (request … via=auto) appears in the log, but with no start time and no result. Consequently, the snapshots for those weeks are likely missing. I cannot see from the log why the job did not start there. Even auto.log only lists the Scrub and Index jobs; the snapshot job is not listed.
• Warning: snaplib.pl only logs a warning for versions below 1.3.0, but still passes the option anyway, which causes the run to abort hard.

How to Fix It

1. Update the cs-freeze4snap tool on member proxmox_1 via About > CS Tools Download (select the member and click Update).
2. Afterwards, start the job manually once and check the log.

I could have found the cause without AI in about 10 minutes, whereas a pure user might have pondered for a long time before finding the cause and simple solution. Of course, one could argue that in a cluster, all members should always be updated rather than just individual ones, or that the job could check program versions. But there are also other OS or storage problems that aren't as straightforward as this one, requiring you to read multiple log files and research where such an issue has been described before. That is precisely where AI with up-to-date data is an unbeatable helper.