Using ReFS for single drives

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

0utf0xZer0

New Member
Aug 17, 2016
7
0
1
37
Hello,

This isn't a server question, but I thought this might be a good place to ask for advice as there aren't a lot of ReFS users on more general hardware forums. As I regularly move between two locations with different desktop computers, I recently bought a 3TB WD Passport to store a bunch of personal files including my Adobe Lightroom catalog.

For reliability reasons, I was thinking of formatting the drive in ReFS, which I understand you can use with a single drive although it's hard to access. I am hoping this will help protect against power loss (always a risk with portable drives as USB 3.0 doesn't seem to be the most reliable connector in my experience) and, if I enable integrity streams, bitrot.

I understand, however, that this comes with a performance and compatibility penalty. Do any of you have advice on whether or not to use ReFS in a situation like mine? Would performance still be sufficient for Lightroom? What about if I add encryption to the mix (since this is a portable drive)?

Any help you can provide would be greatly appreciated.
 

cheezehead

Active Member
Sep 23, 2012
723
175
43
Midwest, US
Is data integrity generally better and there less of a check of needing to run a chkdsk with ReFS? Yes

A second copy of the data would be better.
 

jacobwilliam

New Member
Aug 9, 2016
3
0
1
37
No matter how well a storage array performs, if my data won't be safe from drive failure in a real world scenario, I don't care how fast it's whizzing to and from the storage set. And the business customers we support have the same viewpoint. This is exactly the reason I don't employ RAID-0 arrays in any fashion at customer sites. It's a super fast, but super dangerous, technology to be using when storage resiliency is at stake.

So for my first round of tests, I decided to build out test Storage Spaces that were configured in both mirroring and parity modes. The parity Space was built out with 3 of my disks, using (2) of the Seagate 2TB drives and (1) of the 250GB WD disks I had; the mirror Space was merely using the same sized Seagate 2TB drives. I loaded up a batch of misc test files onto each Storage Space, to represent just a hypothetical set of data that may be sitting on such an array. The test files consisted of misc photos, images, and vector graphics, with some document files mixed in as well.

I then replicated a drive loss on my Spaces by simply hot unplugging the power to one of the disks inside the server, and kept close watch on the signals from Windows about the lost drive. Both Storage Spaces setups worked as advertised. I even gave the system a few reboots to simulate what someone may do in the real world while investigating a lost drive, and tried to access my data on the Spaces respectively during drive loss.
 

0utf0xZer0

New Member
Aug 17, 2016
7
0
1
37
Thank you for the replies.

To clarify: I intend to back up the data on the drive. My concern is that if a file on my main drive becomes corrupt, I may not realize it until after that corrupt file has been copied to the backups. I am hoping that ReFS can help prevent this will still returning usable performance.

I would love to use something more robust than a single USB 3 external drive for my main copy, but I need to be able to fit it in a small bag along with everything else when I travel between locations which seriously limits my options.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
buy 3 * the Seagate 4TB USB 3.0 external drives and do a triple mirror with REFS. then you should not have data corruption while you travel. and it will be relatively small to boot.

Chris
 

nasomi

Member
Jan 11, 2016
47
11
8
40
It feels like no one is answering the guys question. I don't think he wants to build out a whole solution, when all he wants to do is be concerned with the integrity of his single external hdd data.

I think I understand what you're getting at, desiring the "self healing" ability of the ReFS system, which keeps data optimized when it's not being actively used. I think that aspect of the FS is solid, however, I haven't experienced an issue where it would come in to play in NTFS. I haven't heard of any critical issues with ReFS thus far, and have it as the FS of choice on my new test environment. I haven't suffered any speed hits either, everything seems to be operating exactly as it did in NTFS.
 

Keljian

Active Member
Sep 9, 2015
428
71
28
Melbourne Australia
The thing is.. what makes ReFS, Btrfs and ZFS resilient (mostly) is "the scrub". You can't initiate a scrub on ReFS manually, therefore a good part of the benefit of the self healing ability is lost with devices that aren't plugged in all the time.

This: GitHub - maharmstone/btrfs: WinBtrfs could be the answer for you, rather than hacking ReFS to work, if you use windows (which presumably you do otherwise you wouldn't be considering ReFS)
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
The issue is that even with a single disk. all that REFS will only tell you that the data is corrupt (stored checksum <> calculated checksum).

If you need to be sure that you have multiple disks so that there are multiple copies, so that if one is corrupt you have an immediate good copy.

is ~ $200-$300 worth that piece of mind?

Chris
 

ruffy91

Member
Oct 6, 2012
71
11
8
Switzerland
The issue is that even with a single disk. all that REFS will only tell you that the data is corrupt (stored checksum <> calculated checksum).

If you need to be sure that you have multiple disks so that there are multiple copies, so that if one is corrupt you have an immediate good copy.

is ~ $200-$300 worth that piece of mind?

Chris
This was OPs question. Will ReFS detect the corrupt file before (or while) it's backed up. I did not yet see any serious tests which focused on the resiliency of ReFS. Will it detect Bit Error on write? (In other words does it read back the written data). Are any of the resiliency features configurable, if yes what is the optimal configuration, etc.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
There is a daily scan of the ReFS volume. and it compares the stored checksum with the calculated checksum. and that will detect corruption.

on reads it reads the data and the metadata and does the checksum compare. and if there are multiple disks will rescan the other metadata and data until it gets a good copy.

I am not certain if ReFS rescans the file after it is completed being written to the volume to verify the checksum. if it does it would explain why ReFS is slower than NTFS.

Chris
 

0utf0xZer0

New Member
Aug 17, 2016
7
0
1
37
buy 3 * the Seagate 4TB USB 3.0 external drives and do a triple mirror with REFS. then you should not have data corruption while you travel. and it will be relatively small to boot.

Chris
I didn't realize that storage spaces could be used on USB drives but I did a bit of Google-ing and it sounds like it works... I would probably avoid the 4TB 2.5 inch Seagate's though since I understand they use SMR, unlike the 3TB inch WDs.

The issue is that even with a single disk. all that REFS will only tell you that the data is corrupt (stored checksum <> calculated checksum).

If you need to be sure that you have multiple disks so that there are multiple copies, so that if one is corrupt you have an immediate good copy.

is ~ $200-$300 worth that piece of mind?

Chris
I am trying to decide that at the moment.

One side of me is looking and going "who worries about data corruption occurring on an ordinary desktop?".
The other side of me remembers that yeah, I did occasionally find corrupt files on my previous transport drive (a 1.5TB Seagate GoFlex) and I could never figure out why it this would happen. I do recall wondering if it was more prone to bad sectors than my other drives due to extra wear and tear from transport.

Right now my transport drive is a 3TB WD Passport that gets backed up to a 4TB Toshiba X300 on one of my PCs. I have thought about complimenting the passport with a 1TB SSD and external housing - hence why my backup drive is 4TB - but am waiting to see if prices drop further as I am not that desperate for extra performance right now.

Adding another 3TB Passport to use mirroring wouldn't be too pricy but I'm not thrilled about the prospect of having to buy two SSDs if I eventually decide to do such an upgrade..

A couple other question:
1) One happens if one of the drives gets disconnected by accident? I've found even the slightest bump can cause a USB 3 cable to lose it's connection, and while I have run my cabling to minimize this issue it still is a concern.
2) What encryption methods work with storage spaces and ReFS? I know from testing with a single ReFS drive that writing to Veracrypt file containers stored on an ReFS drive is very slow unless I disable the integrity bit on the container, so I am guessing I would need to use full drive encryption?
 

0utf0xZer0

New Member
Aug 17, 2016
7
0
1
37
After giving things some thought I am still leaning towards simply using a single drive with ReFS checksumming. I don't think my data requires automatic replacement, but I do want to know if I'm about to overwrite my backup with a corrupt file.

With that being said, does anyone know what happens when ReFS detects corruption and can't replace it? Does it warn and let you decide whether to access the data, or simply deny access?

I figured out the encryption stuff and related performance issues on my own so that won't be an issue.