To ensure the safety, integrity, and recoverability of data, the following four concepts are essential: Redundancy, Validation, Versioning, and Backup.
1. Redundancy
Redundancy means that a disk failure does not lead to data loss. Work can continue immediately, and the newest version of a file being edited remains available.
2. Validation
Validation means that all data and metadata are stored with checksums (prüfsummen). Data changes/corruption (bit rot) are detected upon reading, and can be automatically repaired if redundancy is present, creating self-healing file systems.
3. Versioning
Versioning means that not only the current data state but also versions at specific points in time are directly available.
4. Backup
Backup means that data remains available, at least in an older state, even in the event of a disaster (hardware failure, fire, theft).
File Systems under Windows
NTFS
NTFS (New Technology File System) has been the standard for many years and is very mature. However, it lacks the two most important options of modern file systems:
ReFS
Microsoft offers ReFS (Resilient File System), which, like ZFS, incorporates Copy-on-Write and Checksums.
The Open Source ZFS file system is now also available for Windows.
Storage Management
1. Redundancy
Redundancy means that a disk failure does not lead to data loss. Work can continue immediately, and the newest version of a file being edited remains available.
- Software RAID Options:
- Normal RAID: Redundancy across entire disks.
- Segmented RAID (e.g., Synology SHR, ZFS AnyRAID soon): Redundancy based on disk segments, allowing full utilization of disks with mixed capacities.
- Windows Storage Spaces: Redundancy based on file copies (not whole disks) across the virtual disks (Spaces).
- Windows Implementation: Redundancy under Windows can be achieved through Hardware RAID, simple Software RAID (Windows Disk Management, mainboard RAID), or Modern Software RAID (Storage Spaces or ZFS).
- Tiering: Storage Spaces also offers an optional hot/cold auto-tiering feature between HDDs and Flash storage.
2. Validation
Validation means that all data and metadata are stored with checksums (prüfsummen). Data changes/corruption (bit rot) are detected upon reading, and can be automatically repaired if redundancy is present, creating self-healing file systems.
- Windows Support: This feature is supported under Windows using ReFS or ZFS.
3. Versioning
Versioning means that not only the current data state but also versions at specific points in time are directly available.
- Mechanism: Modern versioning is extremely efficient, using Copy-on-Write (CoW) procedures based on stored data blocks before a change, instead of making full copies of entire files. This allows for thousands of versions (e.g., one version per hour/last day, one version per day/last month) without excessive overhead.
- Windows Implementation: Versioning is available in Windows using Shadow Copies with NTFS/ReFS or ZFS Snapshots. Access to these versions is usually done via the "Previous Versions" feature or within the file system (read-only ZFS Snap folders).
4. Backup
Backup means that data remains available, at least in an older state, even in the event of a disaster (hardware failure, fire, theft).
- The 3-2-1 Rule: The golden rule for backups is to always have:
- 3 copies of the data.
- On 2 different media/systems.
- With 1 copy stored externally (offsite).
- Process: Backups synchronize the storage with the original data to a backup medium, with or without further versioning on the backup medium.
- Backup Media: Options include another NAS, external drives (including USB), or cloud storage.
- Modern Method (ZFS): ZFS Replication is a modern synchronization method that allows high-load servers with open files to be synchronized with a backup down to a 1-minute delay, even between ZFS servers with different operating systems over the network.
File Systems under Windows
NTFS
NTFS (New Technology File System) has been the standard for many years and is very mature. However, it lacks the two most important options of modern file systems:
- Copy-on-Write (CoW) (needed for crash-safety and snapshots).
- Checksums on data and metadata (needed for Validation/bit-rot protection).
ReFS
Microsoft offers ReFS (Resilient File System), which, like ZFS, incorporates Copy-on-Write and Checksums.
- Status: ReFS has been available since Windows Server 2012 and will soon be available as a boot system.
- Limitations: ReFS still lacks many features found in NTFS or ZFS but is under continuous development.
- Compatibility: ReFS is not backward compatible. Newer ReFS volumes cannot be opened on older Windows versions, which can be problematic with automatic updates.
The Open Source ZFS file system is now also available for Windows.
- Status: The corresponding file system driver for Windows is still in beta (release candidate), making it unsuitable for mission-critical enterprise applications. However, most known bugs under Windows have been resolved, so it's worth a closer look. The issue tracker should be monitored.
Storage Management
- Storage Spaces: Managed using the Windows GUI Tools combined with PowerShell.
- ZFS: Managed via the command-line programs zfs and zpool.
- Alternative Management (GUI): Both Storage Spaces and ZFS can optionally be managed in the browser using a web-GUI like napp-it cs.
- napp-it cs: A portable (Copy and Run) Multi-OS and Multi-Server tool.
- Automation: Tasks can be automated using Windows Scheduled Tasks or napp-it cs jobs.