software raid with real-time protection

[ari2asem]

hi fellow SHT-ers,

my situation now:

using snapraid with windows 10 with 2 data drives and 2 parity drives.
- making backup of 3 software every day
- making par2- files of these 3 backups (25% par2)
- syncing with snapraid every day (snapraid -h...prehashing before syncing)
- scrubing every day (scrub -new)
- scrubing every weekend (scrub -full)

all this actions are done manually. so, it takes about 1 hour to finish it all (from making zip-backup file to scrub -new).

what do i want:

software raid solution for windows 10 with:
- real-time protection (automatuc sync and scrub),
- bit-rot protection (silent error recognition)
- real-time hash and parity checking
- no Linux and no NAS
- software raid running on local machine with windows
- parity protection like snapraid (6 parity disk)
- adding disk to the storage with filled disks like in snapraid

not an option for me:
- MS storage space (no support for silent errors/bit-rot)
- freenas (it is a nas and linux)
- unraid (no file integrity checking)
- hardware raid

any idea?

 

[gea]

what do i want:

software raid solution for windows 10 with:
- real-time protection (automatuc sync and scrub),
- bit-rot protection (silent error recognition)
- real-time hash and parity checking
- no Linux and no NAS
- software raid running on local machine with windows
- parity protection like snapraid (6 parity disk)
- adding disk to the storage with filled disks like in snapraid

not an option for me:
- MS storage space (no support for silent errors/bit-rot)
- freenas (it is a nas and linux)
- unraid (no file integrity checking)
- hardware raid

any idea?

Realtime checksum protection with self healing, crash resistent CopyOnWrite filesystems with snaps and bitrot protection means a nextgen filesystem like ZFS, btrfs or ReFS. On Windows your main option is ReFS + raid via storage spaces. Per default it only offers metadata protection but data protection can be added. Not as easy as ZFS and not as fast or feature rich.

ZFS is on the way for Windows but not ready for production use, openzfsonwindows/ZFSin

btw.
Most advanced ZFS storage appliances are not on Linux but on Unix like Solaris where it comes from or Free-BSD where it is available for some time. Newest ZFS featutures are on Linux and Solaris/OmniOS where it gives a "it just works" experience. But as you said, no NAS or iSCSI Luns (that are offered like local disks on Windows). So you have only "second best" solutions to choose from.

 

[ari2asem]

Realtime checksum protection with self healing, crash resistent CopyOnWrite filesystems with snaps and bitrot protection means a nextgen filesystem like ZFS, btrfs or ReFS. On Windows your main option is ReFS + raid via storage spaces. Per default it only offers metadata protection but data protection can be added. Not as easy as ZFS and not as fast or feature rich.

ZFS is on the way for Windows but not ready for production use, openzfsonwindows/ZFSin

btw.
Most advanced ZFS storage appliances are not on Linux but on Unix like Solaris where it comes from or Free-BSD where it is available for some time. Newest ZFS featutures are on Linux and Solaris/OmniOS where it gives a "it just works" experience. But as you said, no NAS or iSCSI Luns (that are offered like local disks on Windows). So you have only "second best" solutions to choose from.

nas or iscsi is not an option.

ReFS is not supported by windows 10 anymore (nor windows server 2019, nor windows 10 enterprise).

any other options?

 

[i386]

ReFS is not supported by windows server 2019,

Wait... What?

 

[cesmith9999]

There were too many server 2019 storage features in 2019 that are dependent on refs for it to not be supported. I know I was there.

Chris

 

[ari2asem]

suppose this situation...i have 4 disks filled with files. i want to add them to storage space with ReFS under windows server 2019.

can i add filled disks to storage space, without first to format it?

 

[azev]

There's also this

 

[gea]

You can only create a ReFS formatted partition and copy files over from ntfs/fat32 or whatever filesystem you currently have on the disks. There is no conversion option (as far as I know) from ntfs/fat32 to ReFS.

 

[ari2asem]

There's also this

how am i going to use this on a local machine with windows 10 or windows server (2019 or 2016) ?

 

[gea]

btrfs raid above raid-1 is considered unstable even on Linux.
There is currently no serious alterntive to ReFS on Windows if you want a new gen filesystem.

 

[BoredSysadmin]

ReFS 3.4 supported on Windows 10 Pro for Workstations/Enterprise v1803 and newer, also server versions [2019 v1803 or newer]
Source: ReFS - Wikipedia

btw: Freenas isn't build on Linux, but on BSD - similar to OS-X. You don't have to DIY build Freenas, but buy easy to manage FreeNAS Mini from ixsystems - company which develops Freenas
https://www.ixsystems.com/freenas-mini/

 

[ari2asem]

thank you all for suggestions and help.

my conclussion is: there are no solutions for my requirments under windows as i thougth

pretty bad and sad.

the best combination would be zfs and snapraid.
zfs for real-time bit-rot protection,
snapraid for adding hdd's of any size and in any condition and at any time

i would call it SNAPZFS....lol

 

[BoredSysadmin]

just fyi: ZFS does also support disk parity protection, very much equivalent to raid levels.
It also allows "importing" existing ZFS volumes without data loss (limitations apply), but you'd have to do a new copy of data from NTFS to ZFS volume.
There is also a commercial Unraid which support bitrot protection with btrfs. It is very flexible about adding HDDs of any size and mixing different size disks.
@IamSpartacus maybe able to offer some more insight....

 

[IamSpartacus]

Unraid with BTRFS can detect bitrot but not correct it since each disk is it's own file system. That is where backups would come into play. However, I've never seen bitrot in my 10 years of running Unraid. And i'll reference these quotes that should be helpful as well:

I'm using BTRFS with ECC. If a parity check shows an error, I scrub the disks which identifies the corrupted file(s) which can then be recovered from backups. I've never had to do this however, bit rot is rare.

To protect yourself against bitrot, use HDDs manufactured later than year 1994. Its since that time drive's firmware started to add CRC codes to each sector and check them, so any damage to the platter would not go unnoticed, and drive will produce read error instead of faulty data.

In these days, it was a habit of applications that in case of read error, it just skipped over it and replaced data with zero's and went on. But this does not happen anymore: in case of read error, the application is supposed to inform user loud and clear that file went bad.

So

1. Don't use prehistoric 80Mb drives
   
   
2. Don't use weirdo applications that would try to hide that read error happened.
   
   
3. Check important files immediately after they were written. If they went to disk all right, there is no way they will be silently corrupted even if you use FAT32 as file system. If they would get corrupted, it would be loud and clear.
   

ECC memory would protect your data on the way between the place it was produced to the HDD interface. Errors in memory theoretically could silently corrupt your data on the way, but then no filesystem measures would help against it. However, if your data is safely on platter, it would stay that way.

Anyway, the only reliable protection is backup. Anything else is just playing with chances, and in my opinion, that game is not worth the candles.

 

[ari2asem]

but all that (unraid, btrfs, freenas) are not windows.

please, read the OP what i would like to have

no *nix, no nas, no bsd
i want only windows solutions, but they aren't

EDIT: i do make every day offsite backup of 3 backup files and their 25% par2

 

[cesmith9999]

If you add 1 disk to.storage spaces . Run optimize-storagepool. Then you can add space to you virtual disk... then extend your disk in disk manager.


Chris

 

[ari2asem]

If you add 1 disk to.storage spaces . Run optimize-storagepool. Then you can add space to you virtual disk... then extend your disk in disk manager.


Chris

and this could be also done with filled disk?
should i do this for every disk i want to add? adding disks one by one and running commands also one by one?

 

[gregsachs]

and this could be also done with filled disk?
should i do this for every disk i want to add? adding disks one by one and running commands also one by one?

99% sure you can only add empty, uninitialized disks to storage space.
You also need enough disks empty to define a parity space to create a parity space, ie you can't create a space with one disk, add another, put stuff on, add another, make parity; or at least i'm 99% sure of this...
Why not write a .bat file to run the repetitive tasks each day and just email a summary?

 

[ari2asem]

99% sure you can only add empty, uninitialized disks to storage space.
You also need enough disks empty to define a parity space to create a parity space, ie you can't create a space with one disk, add another, put stuff on, add another, make parity; or at least i'm 99% sure of this...
Why not write a .bat file to run the repetitive tasks each day and just email a summary?

good suggestion about to automize the snapraid proccess. there are some scripts for snapraid to do it all with scripts and task sheduler under windows. but making backup of 3 softwares i cann't make them automatic with scripts

 

[ecosse]

I was going to mention FlexRAID but just read the dev seems to have done a runner. I wouldn't have recommended it for a number of reasons but it did hit a lot of the OP requirements as I recollect.

FWIW I've had bad experiences with REFS every time I've used it. Last one was all the volumes going raw - on Windows Server 2016. That was the last time I used REFS

Also FWIW with Snap|RAID and drivepool (which I use) the magical thing to me is that at any point in time I can take that disk away and use it natively.