Drive types / filesystem / method to avoid corrupt data ?
- Thread starter vingo
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Many modern filesystems (ZFS, BTRFS) have built-in protection against data corruption; they can both detect and correct errors caused by the storage media, if given the proper configuration. I use ZFS for all of my important data, including offsite hot and cold backups.
Probably only by testing/verifying the data regularly...
Zfs (the magical silver bullet
), ntfs and other filesystem will happily persist corrupt data they get from an application (I had a somewhat recent problem where mp3tag changed more than id3 tags).
Btw what problem do you try to solve?
For an enterprise the back up strategy and hardware being used could be totally different than what you would need for your plex library at home
Zfs (the magical silver bullet
), ntfs and other filesystem will happily persist corrupt data they get from an application (I had a somewhat recent problem where mp3tag changed more than id3 tags).Btw what problem do you try to solve?
For an enterprise the back up strategy and hardware being used could be totally different than what you would need for your plex library at home
Difficult but doable. Also on a budget.
For starters, avoid corruption of data in the running system by employing stable CPU/RAM/board/software combinations, so there is no garbage in/garbage out. This means the full stack from mature application software, mature filesystem (only ZFS can really compete - checksums everything), mature OS (sorry no ReactOS), mature server hardware and no consumer grade (so Xeon E5, Scalable, EPYC, any ECC-RAM).
Hardware should be verifiable and verified regularly by dedicated software for errors. That can be RAM ECC or CPU errors, so smaller E3 Xeons or Ryzen are out, too (not enough uncore-oomph for diagnostics). Observe SMART data from drives, cooling fan performance, ZFS errors during weekly or monthly scrubs, unexpected kernel errors suddenly appearing in the system logs. All drives are vulnerable to data loss. Beaten-up SSDs from ebay left powered off are the worst. Could see unreadable blocks within a year when left sitting in a hot place. HDDs (SATA, SAS) not even one order of magnitude better, I give them a handful of years until they also grow unreadable sectors, e.g. while sitting in a drawer. Only LTO tapes and quality DVD-R/BD-R last 20-50 years until irrecoverable corruption sets in. But discs are 50 GB max and discerning quality is near impossible. Store tapes vertically and in a cold, dark, and moderately humid place.
Only buy hardware with free and easy to access firmware updates, especially SSDs and NVMEs. There should have been 2-3 updates already so the most eggregious programming errors are corrected. Flash devices without power-loss-protection (PLP) are always the inferior choice and should be avoided. Both items are a nasty source of data corruption if ignored.
Employ aggressive snapshotting with automatic grandfathering (scriptable) so a simple user error, like deleting everything, is easy to correct.
Write backups to disk and then to LTO tape. Google the 3-2-1 backup scheme. Tape to create an airgap between encrypting blackmail trojans and your data. Tape to take stuff offsite in case there is a fire or you get robbed. Backup without verification is just wishful thinking. Verify restore is working once a year.
Consider lightning. One 300kA ("Wilder Hausrüttler") strike into the building will zap everything with a chip. Isolate PoE cable runs, 110/230V mains and DSL/telephone lines with quality protection devices. If fiber is available, get fiber.
Prepare for power outages and UPS failures. APC used to be sturdy, but now they're just another Made in China shop. Eaton the same, currently I like them better still. For UPS failures there could be an automatic transfer switch (ATS) which can switch over to mains within 20 milliseconds should your UPS shutdown due to failure.
Personally I use 2nd gen Xeon Scalables, RDIMM ECC RAM, HGST helium SATA drives, Micron MAX SSDs, Supermicro and Asrock Rack boards, Seasonic power supplies. For storage I have a NetApp DS4246 shelf with redundant LSI 9207 8e controllers. For networking I use Mellanox ethernet cards, cables and switches. Lightning protection by Dehn and some specialty brands.
On the software side I run ZFS on Linux, with pyznap for snapshots, Bareos for backup to LTO tape, rasdaemon, smartd, zed, with comprehensive error reporting by e-mail.
Also sorry for lack of brevity. Didn't have time to write a short answer, so I wrote a long one instead.
For starters, avoid corruption of data in the running system by employing stable CPU/RAM/board/software combinations, so there is no garbage in/garbage out. This means the full stack from mature application software, mature filesystem (only ZFS can really compete - checksums everything), mature OS (sorry no ReactOS), mature server hardware and no consumer grade (so Xeon E5, Scalable, EPYC, any ECC-RAM).
Hardware should be verifiable and verified regularly by dedicated software for errors. That can be RAM ECC or CPU errors, so smaller E3 Xeons or Ryzen are out, too (not enough uncore-oomph for diagnostics). Observe SMART data from drives, cooling fan performance, ZFS errors during weekly or monthly scrubs, unexpected kernel errors suddenly appearing in the system logs. All drives are vulnerable to data loss. Beaten-up SSDs from ebay left powered off are the worst. Could see unreadable blocks within a year when left sitting in a hot place. HDDs (SATA, SAS) not even one order of magnitude better, I give them a handful of years until they also grow unreadable sectors, e.g. while sitting in a drawer. Only LTO tapes and quality DVD-R/BD-R last 20-50 years until irrecoverable corruption sets in. But discs are 50 GB max and discerning quality is near impossible. Store tapes vertically and in a cold, dark, and moderately humid place.
Only buy hardware with free and easy to access firmware updates, especially SSDs and NVMEs. There should have been 2-3 updates already so the most eggregious programming errors are corrected. Flash devices without power-loss-protection (PLP) are always the inferior choice and should be avoided. Both items are a nasty source of data corruption if ignored.
Employ aggressive snapshotting with automatic grandfathering (scriptable) so a simple user error, like deleting everything, is easy to correct.
Write backups to disk and then to LTO tape. Google the 3-2-1 backup scheme. Tape to create an airgap between encrypting blackmail trojans and your data. Tape to take stuff offsite in case there is a fire or you get robbed. Backup without verification is just wishful thinking. Verify restore is working once a year.
Consider lightning. One 300kA ("Wilder Hausrüttler") strike into the building will zap everything with a chip. Isolate PoE cable runs, 110/230V mains and DSL/telephone lines with quality protection devices. If fiber is available, get fiber.
Prepare for power outages and UPS failures. APC used to be sturdy, but now they're just another Made in China shop. Eaton the same, currently I like them better still. For UPS failures there could be an automatic transfer switch (ATS) which can switch over to mains within 20 milliseconds should your UPS shutdown due to failure.
Personally I use 2nd gen Xeon Scalables, RDIMM ECC RAM, HGST helium SATA drives, Micron MAX SSDs, Supermicro and Asrock Rack boards, Seasonic power supplies. For storage I have a NetApp DS4246 shelf with redundant LSI 9207 8e controllers. For networking I use Mellanox ethernet cards, cables and switches. Lightning protection by Dehn and some specialty brands.
On the software side I run ZFS on Linux, with pyznap for snapshots, Bareos for backup to LTO tape, rasdaemon, smartd, zed, with comprehensive error reporting by e-mail.
Also sorry for lack of brevity. Didn't have time to write a short answer, so I wrote a long one instead.
Sort of similar here: ZFS on Linux, main storage is RAID10 (three pairs of two drives), daily snapshots, snapshots replicated to removable drive (one of three that are rotated through an offsite location) and to a dedicated server in a DC many miles away (replication managed using zrepl). smartmon and zed all configured to use email to let me know of issues, and daily journal (log) emails from all machines so I can watch for warnings. No tape here though!
OKAY, thank you very much for the details. I have saved this information, but not on an LTO tape unfortunately
Do not appologise for the"long story", I really appreciate the thoroughness and it gives me a lot of new things to read up on an learn which makes me happy.
I will be starting a little ( a lot) slower, but will use this info to build up.
Questions:
1) Why are commercial boards out ?
2) Was thinking AM4 Asock + ECC for the CPU board combo, what is the issue with it ?
3) Asrock supports unbuffered ECC is this at any major disadvantage for storage corruption prevention ?
4) What kind of drives are "good" enough ... does that exist ?
5) Mirror, multiple drives, different RAIDS, I am lost what is a "good" architecture ?
Problem to solve: VM host/server and important-to-me dataProbably only by testing/verifying the data regularly...
Zfs (the magical silver bullet
Btw what problem do you try to solve?
For an enterprise the back up strategy and hardware being used could be totally different than what you would need for your plex library at home
Like I said newb, but I want to do it right from the start. Any tips welcome as this is overwhelming seeing it for the first time.
There is no single answer; every choice you will make has pros and cons, or tradeoffs, or whatever way you want to look at it.
For example, choosing between RAID1, RAID10, RAID5, and RAID6 is based on the amount of storage you need to be able to use, the amount of raw storage you're willing to lose to redundancy, the read- and write-performance you need for the content you want to store, and probably even more factors.
Also, @Stephan said 'consumer', not 'commercial'. Consumer motherboards often don't offer ECC, don't offer remote management (IPMI or BMC), etc. etc.
As far as ECC and storage corruption, those are completely unrelated. The storage system doesn't make use of the RAM in the system; once data has been sent to the drives they are responsible for it. If the data was uncorrupted at that point, ECC won't make any difference, and if the data was already corrupted at that point then the storage system can't do anything about it. Implementing a system to hold data you care about without using ECC would be madness, but some people do it. Whether the RAM is buffered or unbuffered has no impact on data quality, that affects memory transfer bandwidth.
All modern drives are 'good enough', if a manufacturer made drives that didn't reliably store data under normal conditions they'd be out of business quickly.
For example, choosing between RAID1, RAID10, RAID5, and RAID6 is based on the amount of storage you need to be able to use, the amount of raw storage you're willing to lose to redundancy, the read- and write-performance you need for the content you want to store, and probably even more factors.
Also, @Stephan said 'consumer', not 'commercial'. Consumer motherboards often don't offer ECC, don't offer remote management (IPMI or BMC), etc. etc.
As far as ECC and storage corruption, those are completely unrelated. The storage system doesn't make use of the RAM in the system; once data has been sent to the drives they are responsible for it. If the data was uncorrupted at that point, ECC won't make any difference, and if the data was already corrupted at that point then the storage system can't do anything about it. Implementing a system to hold data you care about without using ECC would be madness, but some people do it. Whether the RAM is buffered or unbuffered has no impact on data quality, that affects memory transfer bandwidth.
All modern drives are 'good enough', if a manufacturer made drives that didn't reliably store data under normal conditions they'd be out of business quickly.
1) You mean consumer boards? Easy: While they have a "supports ECC" sticker on the box, how can you be certain that the CPU is actually performing periodic patrol reads of the RAM, looking for errors? And that the OS will get a report about it? You can't be certain with consumer boards, you need heavy server grade for that. E3 for entry features, E5 formerly, EPYC and Xeon Scalable these days for the best feature set. Board manufacturer has to also test, that this error detection and reporting to the operating system indeed works.
2) See 1.
3) Lower total max capacity compared to RDIMMs, usually 2 channels with 4 sticks not 6 or 8 channels with 2 sticks each on big boards. Possible UDIMMs are more expensive. RDIMM is standard stuff and e.g. DDR4 is available in quantity.
4) Anything m.2 NVME or SATA SSD which has PLP. You need a search engine which allows parametrable searches. Personally I use Micron MAX 5200 and 5300 SATA drives for VMs. Not many simultaneous users though and any databases fit into RAM.
5) ZFS RAIDZ2 if you have 8 large disks or more. Below that, RAIDZ1. If you only have two disks, mirror.
If you are just starting out, don't go overboard, start cheap like with an E3 HP Z240. For something bigger, check out this guy:
vcojot.blogspot.com
vcojot.blogspot.com
vcojot.blogspot.com
The Dell T140 as a frankenstation : Compact, Silent and Powerful enough. if you feel adventurous and like to solder
2) See 1.
3) Lower total max capacity compared to RDIMMs, usually 2 channels with 4 sticks not 6 or 8 channels with 2 sticks each on big boards. Possible UDIMMs are more expensive. RDIMM is standard stuff and e.g. DDR4 is available in quantity.
4) Anything m.2 NVME or SATA SSD which has PLP. You need a search engine which allows parametrable searches. Personally I use Micron MAX 5200 and 5300 SATA drives for VMs. Not many simultaneous users though and any databases fit into RAM.
5) ZFS RAIDZ2 if you have 8 large disks or more. Below that, RAIDZ1. If you only have two disks, mirror.
If you are just starting out, don't go overboard, start cheap like with an E3 HP Z240. For something bigger, check out this guy:
Some Tips about using a Dell PowerEdge T640 as a Workstation (or Frankenstation)
Rationale As much as I liked my T4X0 machines (T430, T440) with RHEL, I started to feel limited in terms of PCIe slots, HDD bays, DIMM slot...
vcojot.blogspot.com
Some Tips about PowerEdge as Workstation (Revisited for 14th Gen servers)
A new computer: Dell PowerEdge T440 server. As much as I consider it a very fine machine now, that road wasn't easy. Some of the previo...
vcojot.blogspot.com
Some Tips about running a Dell PowerEdge Tower Server as your workstation
Some use workstations as servers. I'm using servers as workstations. Over the years, I've changed computing gear on quite a few occasions....
vcojot.blogspot.com
1) You mean consumer boards? Easy: While they have a "supports ECC" sticker on the box, how can you be certain that the CPU is actually performing periodic patrol reads of the RAM, looking for errors? And that the OS will get a report about it? You can't be certain with consumer boards, you need heavy server grade for that. E3 for entry features, E5 formerly, EPYC and Xeon Scalable these days for the best feature set. Board manufacturer has to also test, that this error detection and reporting to the operating system indeed works.
2) See 1.
3) Lower total max capacity compared to RDIMMs, usually 2 channels with 4 sticks not 6 or 8 channels with 2 sticks each on big boards. Possible UDIMMs are more expensive. RDIMM is standard stuff and e.g. DDR4 is available in quantity.
4) Anything m.2 NVME or SATA SSD which has PLP. You need a search engine which allows parametrable searches. Personally I use Micron MAX 5200 and 5300 SATA drives for VMs. Not many simultaneous users though and any databases fit into RAM.
5) ZFS RAIDZ2 if you have 8 large disks or more. Below that, RAIDZ1. If you only have two disks, mirror.
If you are just starting out, don't go overboard, start cheap like with an E3 HP Z240. For something bigger, check out this guy:
Some Tips about using a Dell PowerEdge T640 as a Workstation (or Frankenstation)
Rationale As much as I liked my T4X0 machines (T430, T440) with RHEL, I started to feel limited in terms of PCIe slots, HDD bays, DIMM slot...Some Tips about PowerEdge as Workstation (Revisited for 14th Gen servers)
A new computer: Dell PowerEdge T440 server. As much as I consider it a very fine machine now, that road wasn't easy. Some of the previo...The Dell T140 as a frankenstation : Compact, Silent and Powerful enough. if you feel adventurous and like to solderSome Tips about running a Dell PowerEdge Tower Server as your workstation
Some use workstations as servers. I'm using servers as workstations. Over the years, I've changed computing gear on quite a few occasions....
Read carefully and noted your points about consumer boards / CPUs. So RDIMM + Server grade board w/ E3/E5/EPYC .
Space and more importantly NOISE, the hairdryer-like 70+ db sound will not be wife approved. Any suggestions for something that fits the noise/size sensitivity and will not break the budget (too much) or is that asking for something impossible ?
Had no idea that RDIMM + server grade board is the only way to go. I was really convinced for a minute that a 3900X + as Asrock ECC UDIMM board would do me just fine. I stand corrected and am looking for alternatives now. Not sure if there are any small + quiet + Not the cost of a used car. Open to suggestions, do you have any?
We can't know what 'break the budget' is unless you've told us a budget
Honestly, if your goal is storage and some VMs, without major CPU load, you could just get a Synology or QNAP pre-built NAS box and move on.
Honestly, if your goal is storage and some VMs, without major CPU load, you could just get a Synology or QNAP pre-built NAS box and move on.
Budget is "flexible", but might rephrase the question to in the second hand market what is the cheapest EPYC CPU+Board combo that will get me started as an entry point to ECC RDIMM and other important features ?We can't know what 'break the budget' is unless you've told us a budget
Honestly, if your goal is storage and some VMs, without major CPU load, you could just get a Synology or QNAP pre-built NAS box and move on.
As a former user of ZFS in a professional environment (Datto employee) across a very large fleet of devices, I can emphatically say that a lack of ECC and a stick of RAM that has gone bad can and does result in ZFS corruption in a very reproducible fashion. Reproduce means:
1. I detect corruption (errors at the bottom of `zpool status -v`)
2. I rollback or destroy data to clear the corruption
3. Corruption comes back in live data/newer snapshots
4. Replacing the bad RAM and destroy/rolling back the corruption again results in permanent resolution of the issue.
Use good drives. Scrub your pool regularly. Use ECC. Replicate your data, preferably offsite.
You can use Ryzen and ECC as it is fully supported. I would as soon say buy used E5 because more features, more RAM capacity, maybe cheaper, but it is not a requirement. Memory scrubbing helps to mitigate multi-bit errors proactively before they can occur by detecting single-bit errors and fixing them (the "correcting" part of ECC). Without scrubbing, single bit errors still get fixed when they are detected on read, but it is not a proactive approach. Once a multi-bit error occurs, the system is halted (the "detecting" part of ECC) as it is favorable to stop execution rather than continue with known bad data. Most likely, even with a system that does not have a BMC, you would see a BERT (Boot Error Record Table) entry during startup that would tell you that you ran into a multi-bit error that cause the system to stop.
Section 18.3 Advanced Configuration and Power Interface Specification (uefi.org)
acpi, apei: Add Boot Error Record Table (BERT) support - Patchwork (kernel.org)
Wow. I thought folks were speaking out against Ryzen + ECC as it is unregistered for potential for some other features not available on AM4 boards.
Given what you have said above, Asrock has some great consumer boards like a Pro 4 (supports ECC) that a Ryzen 5950X 16 core could find a nice home in.
Do you reckon that if I went the Ryzen route, instead of "budget" EPYC route that I was asking for guidance on, that I would be missing some critical features and is it a major disadvantage to have unregistered ECC as that is the only kind that Ryzen supports ?
Also What do you mean by "Good Drives", what is the most important features you look for?
Stephan also had a post above highlighting why consumer boards which is Ryzen = bad idea, any thoughts on his comments regarding cpu patroling and truth behind am4 marketing of ECC support?
Last edited:
> Do you reckon that if I went the Ryzen route, instead of "budget" EPYC route that I was asking for guidance on, that I would be missing some critical features and is it a major disadvantage to have unregistered ECC as that is the only kind that Ryzen supports ?
If you are using it for home use, even for data you care about, I see no issue. The downsides to consumer ryzen boards is very limited -- a lack of out of band management--meaning that if the system hangs or goes down, physical intervention would be necessary. Unregistered just costs more. There are some technical differences but it does not jeopardize your data to use unbuffered ecc as opposed to registered ecc.
With EPYC or one of the AM4 Asrock Rack boards that has IPMI, you could remotely intervene if there was a problem that caused a loss of access to the OS --e.g. system hang, network misconfiguration, etc. The more enterprise platforms generally allow you to have better uptime. You're not jeopardizing your data on a consumer platform with ECC, but you may be limiting the uptime reliability, if that makes sense.
If you are using it for home use, even for data you care about, I see no issue. The downsides to consumer ryzen boards is very limited -- a lack of out of band management--meaning that if the system hangs or goes down, physical intervention would be necessary. Unregistered just costs more. There are some technical differences but it does not jeopardize your data to use unbuffered ecc as opposed to registered ecc.
With EPYC or one of the AM4 Asrock Rack boards that has IPMI, you could remotely intervene if there was a problem that caused a loss of access to the OS --e.g. system hang, network misconfiguration, etc. The more enterprise platforms generally allow you to have better uptime. You're not jeopardizing your data on a consumer platform with ECC, but you may be limiting the uptime reliability, if that makes sense.
Thanks for clarifying that is very helpful.
ECC UDIMM is more expensive, Are we not able to put ECC RDIMM into an AM4 and run it unbuffered ? (Just a technical side question about Ram)
> put ECC RDIMM into an AM4 and run it unbuffered
Sadly, no. CPU's IMC communicating with registered DIMMs vs unbuffered is completely different.
Sadly, no. CPU's IMC communicating with registered DIMMs vs unbuffered is completely different.
To be clear when I was referring to 'storage corruption' I was talking about corruption in the storage devices themselves, not on the path to/from them, or in the CPU/RAM/etc. All of these are important things to consider, but I'd classify the larger group as 'data corruption', not 'storage corruption'. In that context the use or non-use of ECC RAM in the machine isn't relevant as it has no effect on the storage devices' ability to properly return uncorrupted data that they had been given to store
Now I am really confused.To be clear when I was referring to 'storage corruption' I was talking about corruption in the storage devices themselves, not on the path to/from them, or in the CPU/RAM/etc. All of these are important things to consider, but I'd classify the larger group as 'data corruption', not 'storage corruption'. In that context the use or non-use of ECC RAM in the machine isn't relevant as it has no effect on the storage devices' ability to properly return uncorrupted data that they had been given to store