Bit rot

[Fritz]

Is it something to be concerned about?

 

[oneplane]

Yes

 

[Stephan]

LTO tape no, stable for decades when stored upright in dry, cool, dark place. DVD-R/W, BD-R maybe. Write BD-R at 1x. Everything else, yes. Mitigation: Use filesystem like ZFS that checksums everything i.e. metadata and data, does regular monthly "scrubs" to detect problems, and can recover.

 

[gea]

Bitrot is not related to good or bad hardware but a statistical phenomenom as it happens by chance over time.This means the larger the storage and the longer the time the more errors you have. The only question may be if a certain media type has more or less errors. There is no media without.

As Stephan said, your only chance to handle bitrot problems is not only to use a good hardware but to accept, detect and repair bitrot problems and the best method is checksumming real data and metadata, use redundancy like raid or copies=2 and check data regularly via a scrub to repair detected errors with ZFS on a rw media or array like a mirror. On a readonly media ex with a ZFS pool on files errors can be detected but not repaired what requires additional copies.

btw
there is a similar problem with RAM. Ram errors are also a statistical problem. ECC RAM can detect and repair these problems. Without ECC it can happen that you write bad data with correct checksums. (Very low propability with good hardware but can happen)

 

[Fritz]

Thanks. I have one zfs box (TrueNAS) and was contemplating building another. Think the project is a go. My current zfs box uses all enterprise grade HW including ECC ram, the new box will also.

 

[Stephan]

Just make sure you get scrub messages from ZFS, i.e. zed daemon is running and reporting. Or whatever TrueNAS has. Even when nothing is found. So you don't end up like Linus from LTT. I scrub monthly, am also using smartd to catch dead drives right away. Fully knowing that the predictive quality of SMART is hit-or-miss. To counter user errors, I found pyznap helpful to create periodic automated read-only snapshots. Finally, always backup. 3-2-1 is of course still the way to go.

 

[BackupProphet]

Yes, I don't know how many times ZFS has catched and fixed, unclean shutdowns, bad memory stick, bad SAS and especially BAD SATA cables, corrupted hard drive sectors, and just a random bit flip that happens a few rare times.

 

[Stephan]

Can't remember which video where Level1 Wendell says "the only filesystem I trust", referring to ZFS. Eliciting a response of great astonishment "Really??" from his guest. Yes, really. I wish Arch would support it out of the box, as root file system, with some optimizations pre-applied, booted through the excellent zfsbootmenu.

 

[i386]

Is it something to be concerned about?

It depends if you run "data scrubs" (zfs wording) or consistency checks (hardware raid wording) regularly

 

[mr44er]

Is it something to be concerned about?

Yes, expect it to happen anytime, anywhere. The bigger the data to move, the bigger the capacity of the media, the higher the chance. Even if you have three perfect copies of your cold backup with generations laying around, monthly checked, you could have a bit flip during the restore, when you need to restore it. So not only bit rot is the enemy. But there are lesser known techniques, that automatically fix this. Nearly every HDD/SSD now should have ECC onboard and active. SMART from Seagate (Raw_Read_Error_Rate and Seek_Error_Rate) is often misinterpreted, but that's the internal error correction and everything is fine with this disk.
This one is a fresh (and not refurbished) SATA and has its first 99 hours runtime:

ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   109   100   006    -    21527784            <=======
  3 Spin_Up_Time            PO---K   100   100   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    1
  5 Reallocated_Sector_Ct   PO--CK   100   100   036    -    0
  7 Seek_Error_Rate         POSR-K   063   060   030    -    2544450             <=======
  9 Power_On_Hours          -O--CK   100   100   000    -    99
10 Spin_Retry_Count        PO--CK   100   100   097    -    0
12 Power_Cycle_Count       -O--CK   100   100   020    -    1
180 Unknown_HDD_Attribute   PO-R-K   100   100   000    -    2165301594
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        PO--CK   100   100   097    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    0 0 0
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   068   068   045    -    32 (Min/Max 25/32)
194 Temperature_Celsius     -O---K   032   040   000    -    32 (0 19 0 0 0)
195 Hardware_ECC_Recovered  -O-RCK   041   041   000    -    21527784            <=======

To avoid bit rot, you want to read your data 100% as it was written, no compromise. A single disk has the ability to re-read a sector if necessary a multiple times and ECC and/or based on a guess. If this (injected) guess also fails the checksum, this single disk delivers wrong data. So ZFS is on another layer and does basically the same, but it looks on all disks. The more available checksums (in form of disk count) it has to work with or to make a decision in the case of an error, the better. So think about that first, when planning your pool/disksize/-count and not "meh, raidz2 or -3 is bad, because it swallows all the capacity". It's not only useful when a disk dies completely, it secures your data right from the start.
It is also good to know the opposite fact: ZFS on a single disk is possible, but it can't correct an error, only detect.

More explanation and some visualisation what it really means with example of a picture: Data degradation - Wikipedia
A corrupt video file will stutter, have distorted colors or stop during playback.
A simple text file will look like this:
Good:

This is a sentence in my text file.

Corrupt example(unpredictable):

This is % & &tence in my text^`le.

If you stumble over files and notice that they are corrupt in some way, then it doesn't mean it happened on your hardware. It could be broken already on the source (friend's disk or usbstick) or it could happen during the download. Pausing/resuming downloads (same for 24hour-kick on dial-up lines) provokes this and this is the reason why mostly linux or other .isos have a file with a checksum in the same folder. If the checksum matches, all good. If not or you don't care, you could also burn this image, boot and (try) install. At some point, you will get error or crashes you can't explain.

There is no media without.

+1
Yup, it's a matter of when, not if.

Thanks. I have one zfs box (TrueNAS) and was contemplating building another. Think the project is a go. My current zfs box uses all enterprise grade HW including ECC ram, the new box will also.

That is very good, but even the highest quality stuff is no guarantee to prevent, it can only minimize the risk. A good rule of thumb is scrubbing every month on consumer disks, every three on enterprise grade, everything between is good. As this means a heavy load on all your disks, pay attention that scrubbing too often will statistically kill your disks earlier. Scrubbing every week would be overkill.

Bonus (for SAS disks):
These have modepages to set how they should deal with errors in detail. Linux/TrueNAS/FreeBSD have sdparm as package or plugin to view and edit these modepages.
To show what a disk has set with explanation: sdparm -a --long /dev/sd$
cha: y means, it's changeable, def: 0 shows the vendor/firmware default, sav: 0 shows the saved entry. Saved entries survive a drive restart.
AWRE and ARRE are the settings for r/w reallocation. You really want this, it should be activated as default anyway. If not:
sdparm -s AWRE=1 /dev/sd$ activates, but not save the setting. A cold reboot of the drive or power outage resets this.
So sdparm -s AWRE=1 -S /dev/sd$ sets and saves this.
Or for both: sdparm -s AWRE=1,ARRE=1 -S /dev/sd$
To that function dependent good settings are RRC=8,WRC=8. This means 8 retries for fail at read or write, after that report fail to the upper layer -> controller/ZFS . On mission critical servers/big arrays lesser retries (and shorter timeouts) are desirable, because it provokes ZFS to throw out the disk earlier -> "better to throw out one of the 192 disks/pull the spare than to try to force this disk to read or write this one sector forever and have a sluggish, hanging pool where our webserver runs."
For homelab 8 is ok, maybe 12 too...and over 20 I would only use if I have a single disk with important data...but that is bad in general and if a disk can't repair in some retries, then it can be declared as dead and done.

There's a ton more of the modepages, but don't fiddle with settings you don't understand...your pool and your data will regret wrong settings.
Some more:
WCE=0 I know, most guys here want write cache activated or recommend it. This is ok, good and useful to max. out benchmarks or you mostly write big files onto your array, need the performance and have an UPS.
On the other hand it is much safer for your data in the case of a crash or power outage, when it happens during a write operation. If you mostly read from your array, you likely won't notice a difference.

Another feature I like very much is the background medium scan (BMS). It's like a scrub the disk does for itself during idle times (you won't notice, no performance impact) and it stops immediately when normal operation is demanded. When idle again, it resumes. If errors are found, it tries to fix it (AWRE,ARRE) and if that was successful, all good. If not, ZFS takes over.
Enable it: sdparm -s EN_BMS=1 -S /dev/sd$
Interval in hours to start a scan: sdparm -s BMS_I=168 -S /dev/sd$
168 hours are one week and are default on most generic HGSTs. It runs with low priority and I guess with some secret sauce of the vendor, so it should not crunch up the disk life vs. a ZFS scrub, which is high priority. Speaking of vendor...some disks/firmware have a min (and max?) for this. If it does not work, try a value with +24 or -24.
You can watch the progress and the log (the log appears only after the first error, which from my experience is mostly after the first 20000h disk life or if you explicitly set the log-modepage for everything) with smartctl -x /dev/sd$

Some more details: https://www.reddit.com/r/DataHoarder/comments/e677p1

 

[Whaaat]

default value for background medium scan interval for HGST He10 is 1008 hours. Did they found that 168 hours is a bit of overkill?

If errors are found, it tries to fix it (AWRE,ARRE) and if that was successful, all good

It depends on the vendor realization. I've had bad experience with Seagate. Maybe that's why storage vendors such as Netapp and EMC do not rely on BMS but on PI format instead and controller-based scrubbing.

Here is the log from dying Seagate that succesfully reallocated 3700+ bad sectors while idling in BMS, but the data from reallocated sectors were permanently lost anyway. Dumb drive even throws ASC/ASCQ 18h/04h which is invalid for direct‑access block device (e.g., magnetic disk) , maybe that's why controller chose to ignore those exceptions instead of raising a red flag.

Spoiler: 'Successfully reassigned'

C:\Program Files\smartmontools\bin>smartctl.exe -x pd3
smartctl 7.1 2019-12-30 r5022 [x86_64-w64-mingw32-w10-1903] (sf-7.1-1)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               IBM-ESXS
Product:              ST3000NM0035  E
Revision:             EC83
Compliance:           SPC-4
User Capacity:        3 000 592 982 016 bytes [3,00 TB]
Logical block size:   512 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000c50093a8f817
Serial number:        xxxxxxxxxxx
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat May 20 13:38:27 2023
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Grown defects during certification = 0
Total blocks reassigned during format = 0
Total new blocks reassigned = 3556
Power on minutes since format = 411643
Current Drive Temperature:     37 C
Drive Trip Temperature:        65 C

Elements in grown defect list: 3724

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   474813002        2         0  474813004          2       8104,712           0
write:         0        0         0         0          0       1547,955           0
verify: 2339712267        5         0  2339712272          5      79253,633           0

Non-medium error count:     2704

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   17868                 - [-   -    -]
# 2  Background short  Completed                   -   17867                 - [-   -    -]
# 3  Background short  Completed                   -   17867                 - [-   -    -]
# 4  Background short  Completed                   -   17867                 - [-   -    -]
# 5  Background short  Completed                   -   17867                 - [-   -    -]

Long (extended) Self-test duration: 19020 seconds [317,0 minutes]

Background scan results log
  Status: no scans active
    Accumulated power on time, hours:minutes 18162:35 [1089755 minutes]
    Number of background scans performed: 587,  scan progress: 39,82%
    Number of background medium scans performed: 587

   #  when        lba(hex)    [sk,asc,ascq]    reassign_status
   1 17610:57  000000011a24ad78  [1,18,4]   Recovered via rewrite in-place
   2 17610:57  000000011a24ad79  [1,18,4]   Recovered via rewrite in-place
   3 17610:57  000000011a24ad7a  [1,18,4]   Successfully reassigned
   4 17610:57  000000011a24ad7b  [1,18,4]   Recovered via rewrite in-place
   5 17610:57  000000011a2a7b77  [1,18,4]   Recovered via rewrite in-place
   6 17610:57  000000011a2a7b78  [1,18,4]   Successfully reassigned
   7 17610:57  000000011a325b8e  [1,17,1]   Recovered via rewrite in-place
   8 17610:57  000000011a325b94  [1,18,4]   Recovered via rewrite in-place
   9 17610:57  000000011a4d7523  [1,18,4]   Successfully reassigned
  10 17610:57  000000011a53e70c  [1,18,4]   Successfully reassigned
  11 17610:57  000000011a53e70e  [1,18,4]   Successfully reassigned
  12 17610:57  000000011a53e718  [1,18,4]   Recovered via rewrite in-place
  13 17610:57  000000011a5b23b6  [1,18,4]   Successfully reassigned
  14 17610:57  000000011a5b23ba  [1,18,4]   Successfully reassigned
  15 17610:58  000000011a60ca56  [1,18,4]   Successfully reassigned
  16 17610:58  000000011a60ca57  [1,18,4]   Successfully reassigned
  17 17610:58  000000011a763d4d  [1,18,4]   Successfully reassigned
  18 17610:58  000000011adaccab  [1,18,4]   Successfully reassigned
  19 17610:58  000000011afdc00f  [1,18,4]   Successfully reassigned
  20 17610:58  000000011afdc014  [1,18,4]   Successfully reassigned
  21 17610:58  000000011afdc03a  [1,18,4]   Successfully reassigned
  22 17610:59  000000011b7352bc  [1,17,1]   Successfully reassigned
  23 17611:00  000000011b7352bd  [1,18,4]   Successfully reassigned
  24 17611:00  000000011b7352be  [1,17,1]   Successfully reassigned
  25 17611:00  000000011b7352ca  [1,17,1]   Successfully reassigned
  26 17611:00  000000011b7352cb  [1,18,4]   Successfully reassigned
  27 17611:00  000000011b7352cc  [1,18,4]   Successfully reassigned
  28 17611:00  000000011b7352cd  [1,18,4]   Successfully reassigned
  29 17611:00  000000011b7352ce  [1,18,4]   Recovered via rewrite in-place
  30 17611:00  000000011b7352cf  [1,18,4]   Recovered via rewrite in-place
  31 17611:00  000000011b7352d3  [1,17,1]   Successfully reassigned
  32 17611:00  000000011b7352d4  [1,18,4]   Recovered via rewrite in-place
  33 17611:00  000000011b7352d5  [1,18,4]   Recovered via rewrite in-place
  34 17611:00  000000011b7352d6  [1,18,4]   Recovered via rewrite in-place
  35 17611:00  000000011b7352ed  [1,18,4]   Recovered via rewrite in-place
  36 17611:00  000000011b7352ef  [1,18,4]   Successfully reassigned
  37 17611:00  000000011b7352f0  [1,18,4]   Recovered via rewrite in-place
  38 17611:00  000000011b7352f1  [1,18,4]   Successfully reassigned
  39 17611:00  000000011b7352f2  [1,18,4]   Successfully reassigned
  40 17611:00  000000011b7352f3  [1,18,4]   Recovered via rewrite in-place
  41 17611:00  000000011b7352f4  [1,18,4]   Recovered via rewrite in-place
  42 17611:00  000000011b7352f5  [1,18,4]   Successfully reassigned
  43 17611:00  000000011b7352f6  [1,18,4]   Recovered via rewrite in-place
  44 17611:00  000000011b7352f8  [1,18,4]   Successfully reassigned
  45 17611:00  000000011b7352f9  [1,18,4]   Successfully reassigned
  46 17611:00  000000011b7352fa  [1,18,4]   Successfully reassigned
  47 17611:00  000000011b7352fb  [1,18,4]   Successfully reassigned
  48 17611:00  000000011b7352fc  [1,18,4]   Recovered via rewrite in-place
  49 17611:00  000000011b7352fd  [1,18,4]   Successfully reassigned
  50 17611:00  000000011b7352fe  [1,18,4]   Successfully reassigned
  51 17611:00  000000011b735301  [1,18,4]   Successfully reassigned
  52 17611:00  000000011b735302  [1,18,4]   Successfully reassigned
  53 17611:00  000000011b735304  [1,18,4]   Recovered via rewrite in-place
  54 17611:01  000000011b735306  [1,18,4]   Successfully reassigned
  55 17611:01  000000011b735307  [1,18,4]   Successfully reassigned
  56 17611:01  000000011b735308  [1,18,4]   Recovered via rewrite in-place
  57 17611:01  000000011b735309  [1,18,4]   Successfully reassigned
  58 17611:01  000000011b73530a  [1,17,1]   Successfully reassigned
  59 17611:01  000000011b73530b  [1,18,1]   Successfully reassigned
  60 17611:01  000000011b735311  [1,18,4]   Recovered via rewrite in-place
  61 17611:01  000000011b735312  [1,18,4]   Successfully reassigned
  62 17611:01  000000011b735313  [1,18,4]   Recovered via rewrite in-place
  63 17611:01  000000011b735314  [1,18,4]   Successfully reassigned
  64 17611:01  000000011b735315  [1,18,4]   Successfully reassigned
  65 17611:01  000000011b735317  [1,18,4]   Recovered via rewrite in-place
  66 17611:01  000000011b735318  [1,18,4]   Successfully reassigned
  67 17611:01  000000011b73531b  [1,18,4]   Successfully reassigned
  68 17611:01  000000011b73531d  [1,18,4]   Recovered via rewrite in-place
  69 17611:01  000000011b73531e  [1,18,4]   Recovered via rewrite in-place
  70 17611:01  000000011b73531f  [1,18,4]   Recovered via rewrite in-place
  71 17611:01  000000011b735321  [1,18,4]   Successfully reassigned
  72 17611:01  000000011b735322  [1,18,4]   Successfully reassigned
  73 17611:01  000000011b78f4f9  [1,18,4]   Successfully reassigned
  74 17611:01  000000011bb78b36  [1,18,4]   Successfully reassigned
  75 17611:01  000000011bb78b41  [1,18,4]   Successfully reassigned
  76 17611:01  000000011bb78b42  [1,18,4]   Successfully reassigned
  77 17611:01  000000011bc44384  [1,18,4]   Successfully reassigned
  78 17611:01  000000011bd1c1ba  [1,18,4]   Recovered via rewrite in-place
  79 17611:02  000000011c65ef89  [1,18,4]   Successfully reassigned
  80 17611:03  000000011ce50416  [1,18,4]   Successfully reassigned
  81 17611:03  000000011ce53cb5  [1,18,4]   Successfully reassigned
  82 17611:03  000000011ce53cb6  [1,17,1]   Recovered via rewrite in-place
  83 17611:03  000000011ce53cb7  [1,18,4]   Successfully reassigned
  84 17611:03  000000011ce53cb8  [1,18,4]   Recovered via rewrite in-place
  85 17611:03  000000011ce53cbc  [1,18,4]   Successfully reassigned
  86 17611:03  000000011ce53cc1  [1,18,4]   Successfully reassigned
  87 17611:03  000000011ce53cc6  [1,18,4]   Successfully reassigned
  88 17611:03  000000011ce53cc7  [1,18,4]   Successfully reassigned
  89 17611:03  000000011ce53ccb  [1,18,4]   Successfully reassigned
  90 17611:03  000000011ce53ccd  [1,18,4]   Recovered via rewrite in-place
  91 17611:03  000000011ce5c38c  [1,18,4]   Successfully reassigned
  92 17611:03  000000011ce5c38d  [1,18,4]   Recovered via rewrite in-place
  93 17611:03  000000011ce5c393  [1,17,1]   Successfully reassigned
  94 17611:03  000000011cf26a35  [1,18,4]   Successfully reassigned
  95 17611:03  000000011d0865e1  [1,18,4]   Successfully reassigned
  96 17611:03  000000011d0d9afe  [1,18,4]   Successfully reassigned
  97 17611:03  000000011d0d9aff  [1,18,4]   Recovered via rewrite in-place
  98 17611:03  000000011d153d64  [1,18,4]   Successfully reassigned
  99 17611:03  000000011d1b13fc  [1,18,4]   Recovered via rewrite in-place
 100 17611:03  000000011d1b8ec7  [1,18,4]   Recovered via rewrite in-place
 101 17611:03  000000011d1b8ec8  [1,18,4]   Successfully reassigned
 102 17611:03  000000011d1b8ed8  [1,18,4]   Successfully reassigned
 103 17611:03  000000011d1b8ed9  [1,18,4]   Successfully reassigned
 104 17611:03  000000011d1b8edc  [1,18,4]   Recovered via rewrite in-place
 105 17611:03  000000011d1b8ee9  [1,18,4]   Recovered via rewrite in-place
 106 17611:03  000000011d1b8eea  [1,18,4]   Successfully reassigned
 107 17611:03  000000011d1b8eeb  [1,18,4]   Recovered via rewrite in-place
 108 17611:03  000000011d1b8eec  [1,18,4]   Recovered via rewrite in-place
 109 17611:04  000000011d1b8eed  [1,18,4]   Successfully reassigned
 110 17611:04  000000011d1b8eee  [1,18,4]   Recovered via rewrite in-place
 111 17611:04  000000011d1b8ef6  [1,18,4]   Recovered via rewrite in-place
 112 17611:04  000000011d1b8ef7  [1,18,4]   Successfully reassigned
 113 17611:04  000000011d1b8efd  [1,18,4]   Recovered via rewrite in-place
 114 17611:04  000000011d1b8f00  [1,18,4]   Successfully reassigned
 115 17611:04  000000011d1b8f01  [1,18,4]   Recovered via rewrite in-place
 116 17611:04  000000011d1b8f02  [1,18,4]   Recovered via rewrite in-place
 117 17611:04  000000011d22dabf  [1,18,4]   Recovered via rewrite in-place
 118 17611:04  000000011d22dac5  [1,18,4]   Successfully reassigned
 119 17611:04  000000011d235ef4  [1,18,4]   Recovered via rewrite in-place
 120 17611:04  000000011d235ef5  [1,18,4]   Successfully reassigned
 121 17611:04  000000011d235ef6  [1,18,4]   Recovered via rewrite in-place
 122 17611:04  000000011d235efd  [1,18,4]   Recovered via rewrite in-place
 123 17611:04  000000011d235efe  [1,18,4]   Successfully reassigned
 124 17611:04  000000011d235eff  [1,18,4]   Recovered via rewrite in-place
 125 17611:04  000000011d28bb85  [1,18,4]   Successfully reassigned
 126 17611:04  000000011d297d52  [1,18,4]   Successfully reassigned
 127 17611:04  000000011d297d53  [1,18,4]   Successfully reassigned
 128 17611:04  000000011d297d54  [1,18,4]   Successfully reassigned
 129 17611:04  000000011d297d55  [1,17,1]   Successfully reassigned
 130 17611:04  000000011d297d56  [1,18,4]   Successfully reassigned
 131 17611:04  000000011d297d57  [1,18,4]   Recovered via rewrite in-place
 132 17611:04  000000011d297d58  [1,18,4]   Recovered via rewrite in-place
 133 17611:04  000000011d297d5f  [1,18,4]   Recovered via rewrite in-place
 134 17611:04  000000011d297d6d  [1,18,4]   Successfully reassigned
 135 17611:04  000000011d297d72  [1,18,4]   Recovered via rewrite in-place
 136 17611:04  000000011d297d73  [1,18,4]   Recovered via rewrite in-place
 137 17611:04  000000011d297d74  [1,18,4]   Successfully reassigned
 138 17611:04  000000011d297d76  [1,18,4]   Recovered via rewrite in-place
 139 17611:04  000000011d297d77  [1,18,4]   Successfully reassigned
 140 17611:04  000000011d297d78  [1,18,4]   Successfully reassigned
 141 17611:04  000000011d297d79  [1,18,4]   Successfully reassigned
 142 17611:04  000000011d297d7b  [1,18,4]   Successfully reassigned
 143 17611:04  000000011d297d7c  [1,18,4]   Successfully reassigned
 144 17611:04  000000011d297d7d  [1,18,4]   Successfully reassigned
 145 17611:04  000000011d297d7e  [1,18,4]   Recovered via rewrite in-place
 146 17611:04  000000011d297d80  [1,18,4]   Successfully reassigned
 147 17611:04  000000011d297d81  [1,18,4]   Successfully reassigned
 148 17611:04  000000011d297d82  [1,18,4]   Successfully reassigned
 149 17611:04  000000011d306f42  [1,18,4]   Recovered via rewrite in-place
 150 17611:04  000000011d306f44  [1,18,4]   Recovered via rewrite in-place
 151 17611:04  000000011d306f46  [1,18,4]   Successfully reassigned
 152 17611:04  000000011d306f48  [1,18,4]   Successfully reassigned
 153 17611:04  000000011d36e576  [1,18,4]   Successfully reassigned
 154 17611:04  000000011d36e577  [1,18,4]   Successfully reassigned
 155 17611:04  000000011d36e57b  [1,18,4]   Successfully reassigned
 156 17611:05  000000011d3ea427  [1,18,4]   Recovered via rewrite in-place
 157 17611:05  000000011d3ea428  [1,17,1]   Recovered via rewrite in-place
 158 17611:05  000000011d3ea429  [1,18,4]   Recovered via rewrite in-place
 159 17611:05  000000011d3ea42a  [1,18,4]   Successfully reassigned
 160 17611:05  000000011d3ea42c  [1,18,4]   Recovered via rewrite in-place
 161 17611:05  000000011d3ea42e  [1,18,4]   Successfully reassigned
 162 17611:05  000000011d3ea42f  [1,18,4]   Recovered via rewrite in-place
 163 17611:05  000000011d3ea430  [1,18,4]   Recovered via rewrite in-place
 164 17611:05  000000011d3ea431  [1,18,4]   Recovered via rewrite in-place
 165 17611:05  000000011d3ea432  [1,18,4]   Recovered via rewrite in-place
 166 17611:05  000000011d3ea433  [1,18,4]   Successfully reassigned
 167 17611:05  000000011d3ea434  [1,18,4]   Successfully reassigned
 168 17611:05  000000011d3ea435  [1,18,4]   Successfully reassigned
 169 17611:05  000000011d3ea437  [1,18,4]   Successfully reassigned
 170 17611:05  000000011d3ea438  [1,18,4]   Recovered via rewrite in-place
 171 17611:05  000000011d3ea439  [1,18,4]   Successfully reassigned
 172 17611:05  000000011d3ea43a  [1,18,4]   Successfully reassigned
 173 17611:05  000000011d3ea43b  [1,18,4]   Successfully reassigned
 174 17611:05  000000011d3ea43c  [1,18,4]   Successfully reassigned
 175 17611:05  000000011d3ea43e  [1,18,4]   Successfully reassigned
 176 17611:05  000000011d3ea43f  [1,18,4]   Successfully reassigned
 177 17611:05  000000011d3ea440  [1,18,4]   Successfully reassigned
 178 17611:05  000000011d3ea443  [1,18,4]   Successfully reassigned
 179 17611:05  000000011d3ea449  [1,18,4]   Successfully reassigned
 180 17611:05  000000011d3ea44b  [1,18,4]   Recovered via rewrite in-place
 181 17611:05  000000011d512357  [1,18,4]   Successfully reassigned
 182 17611:05  000000011d51235a  [1,18,4]   Successfully reassigned
 183 17611:05  000000011d51235b  [1,18,4]   Recovered via rewrite in-place
 184 17611:05  000000011d516eec  [1,18,4]   Recovered via rewrite in-place
 185 17611:05  000000011d516eed  [1,18,4]   Successfully reassigned
 186 17611:05  000000011d516eee  [1,18,4]   Recovered via rewrite in-place
 187 17611:05  000000011d516ef1  [1,18,4]   Recovered via rewrite in-place
 188 17611:05  000000011d52353d  [1,18,4]   Recovered via rewrite in-place
 189 17611:05  000000011d52353e  [1,17,1]   Recovered via rewrite in-place
 190 17611:05  000000011d52353f  [1,18,4]   Successfully reassigned
 191 17611:05  000000011d5e9068  [1,18,4]   Successfully reassigned
 192 17611:05  000000011d5e9069  [1,18,4]   Successfully reassigned
 193 17611:05  000000011d5eab8d  [1,18,4]   Successfully reassigned
 194 17611:05  000000011d5eab98  [1,18,4]   Recovered via rewrite in-place
 195 17611:05  000000011d5eabaf  [1,18,4]   Recovered via rewrite in-place
 196 17611:05  000000011d5eabb0  [1,18,4]   Successfully reassigned
 197 17611:05  000000011d5eabb3  [1,18,4]   Successfully reassigned
 198 17611:05  000000011d5eabb4  [1,18,4]   Successfully reassigned
 199 17611:05  000000011d5eabb5  [1,18,4]   Recovered via rewrite in-place
 200 17611:05  000000011d5eabb6  [1,18,4]   Successfully reassigned
 201 17611:05  000000011d5eabbd  [1,18,4]   Successfully reassigned
 202 17611:05  000000011d5eabc0  [1,18,4]   Successfully reassigned
 203 17611:05  000000011d5eabc1  [1,18,4]   Recovered via rewrite in-place
 204 17611:06  000000011d5eabc2  [1,18,4]   Successfully reassigned
 205 17611:06  000000011d5eabc5  [1,17,1]   Recovered via rewrite in-place
 206 17611:06  000000011d5eacaf  [1,18,4]   Successfully reassigned
 207 17611:06  000000011d5eacb0  [1,18,4]   Successfully reassigned
 208 17611:06  000000011d5eacb1  [1,18,4]   Successfully reassigned
 209 17611:06  000000011d5eacb2  [1,18,4]   Successfully reassigned
 210 17611:06  000000011d5eacba  [1,18,4]   Successfully reassigned
 211 17611:06  000000011d5eacbb  [1,18,4]   Recovered via rewrite in-place
 212 17611:06  000000011d5eacbc  [1,18,4]   Recovered via rewrite in-place
 213 17611:06  000000011d5eacbd  [1,18,4]   Recovered via rewrite in-place
 214 17611:06  000000011d5eacbf  [1,18,4]   Successfully reassigned
 215 17611:06  000000011d5eacc0  [1,18,4]   Recovered via rewrite in-place
 216 17611:06  000000011d5eaccd  [1,18,4]   Successfully reassigned
 217 17611:06  000000011d5eacce  [1,18,4]   Recovered via rewrite in-place
 218 17611:06  000000011d5f98cc  [1,18,4]   Successfully reassigned
 219 17611:06  000000011d5f98cd  [1,18,4]   Successfully reassigned
 220 17611:06  000000011d5f98cf  [1,18,4]   Recovered via rewrite in-place
 221 17611:06  000000011d5f98d0  [1,18,4]   Recovered via rewrite in-place
 222 17611:06  000000011d5f98d1  [1,18,4]   Successfully reassigned
 223 17611:06  000000011d5f98d2  [1,18,4]   Successfully reassigned
 224 17611:06  000000011d5f98d3  [1,18,4]   Successfully reassigned
 225 17611:06  000000011d5f98d4  [1,18,4]   Successfully reassigned
 226 17611:06  000000011d5f98d5  [1,18,4]   Successfully reassigned
 227 17611:06  000000011d5f98d6  [1,18,4]   Recovered via rewrite in-place
 228 17611:06  000000011d5f98d7  [1,18,4]   Recovered via rewrite in-place
 229 17611:06  000000011d5f98d9  [1,18,4]   Recovered via rewrite in-place
 230 17611:06  000000011d5f98db  [1,18,4]   Successfully reassigned
 231 17611:06  000000011d664591  [1,18,4]   Successfully reassigned
 232 17611:06  000000011d66459b  [1,18,4]   Successfully reassigned
 233 17611:06  000000011d66459d  [1,18,4]   Successfully reassigned
 234 17611:06  000000011d66459e  [1,18,4]   Recovered via rewrite in-place
 235 17611:06  000000011d66459f  [1,18,4]   Recovered via rewrite in-place
 236 17611:06  000000011d6645a0  [1,18,4]   Recovered via rewrite in-place
 237 17611:06  000000011d6645a1  [1,18,4]   Successfully reassigned
 238 17611:06  000000011d6645a3  [1,18,4]   Recovered via rewrite in-place
 239 17611:06  000000011d6645a7  [1,18,4]   Successfully reassigned
 240 17611:06  000000011d6645a8  [1,18,4]   Successfully reassigned
 241 17611:06  000000011d6645a9  [1,18,4]   Successfully reassigned
 242 17611:06  000000011d6645ab  [1,18,4]   Recovered via rewrite in-place
 243 17611:06  000000011d66a3fa  [1,18,4]   Recovered via rewrite in-place
 244 17611:06  000000011d66a40f  [1,18,4]   Successfully reassigned
 245 17611:06  000000011d66a413  [1,18,4]   Recovered via rewrite in-place
 246 17611:06  000000011d66a540  [1,18,4]   Successfully reassigned
 247 17611:06  000000011d66a541  [1,17,1]   Successfully reassigned
 248 17611:06  000000011d66a544  [1,18,4]   Successfully reassigned
 249 17611:06  000000011d74bc29  [1,18,4]   Successfully reassigned
 250 17611:07  000000011d8ec9a9  [1,18,4]   Successfully reassigned
 251 17611:07  000000011daca5ce  [1,18,4]   Successfully reassigned
 252 17611:07  000000011daca5d3  [1,17,1]   Successfully reassigned
 253 17611:07  000000011db472a4  [1,18,4]   Recovered via rewrite in-place
 254 17611:07  000000011db472a5  [1,18,4]   Successfully reassigned
 255 17611:07  000000011db472a6  [1,18,4]   Recovered via rewrite in-place
 256 17611:07  000000011db472a7  [1,18,4]   Successfully reassigned
 257 17611:07  000000011db472a8  [1,18,4]   Successfully reassigned
 258 17611:07  000000011db472ac  [1,18,4]   Recovered via rewrite in-place
 259 17611:07  000000011db472ad  [1,18,4]   Successfully reassigned
 260 17611:07  000000011db825e5  [1,18,4]   Successfully reassigned
 261 17611:07  000000011dc01e87  [1,18,4]   Successfully reassigned
 262 17611:07  000000011de83802  [1,17,1]   Successfully reassigned
 263 17611:07  000000011de83804  [1,18,4]   Recovered via rewrite in-place
 264 17611:07  000000011de83805  [1,18,4]   Recovered via rewrite in-place
 265 17611:07  000000011de83806  [1,18,4]   Successfully reassigned
 266 17611:07  000000011df51fa1  [1,18,4]   Successfully reassigned
 267 17611:07  000000011df51fa2  [1,17,1]   Successfully reassigned
 268 17611:08  000000011e15110d  [1,18,4]   Recovered via rewrite in-place
 269 17611:08  000000011e15110f  [1,18,4]   Recovered via rewrite in-place
 270 17611:08  000000011e151110  [1,18,4]   Recovered via rewrite in-place
 271 17611:08  000000011e151112  [1,18,4]   Successfully reassigned
 272 17611:08  000000011e151113  [1,18,4]   Recovered via rewrite in-place
 273 17611:08  000000011e1cb7e0  [1,18,4]   Recovered via rewrite in-place
 274 17611:08  000000011e1cb7f0  [1,18,4]   Recovered via rewrite in-place
 275 17611:08  000000011e1cb7f3  [1,18,4]   Recovered via rewrite in-place
 276 17611:08  000000011e1cb7f4  [1,18,4]   Successfully reassigned
 277 17611:08  000000011e1cb7f5  [1,18,4]   Recovered via rewrite in-place
 278 17611:08  000000011e1cb7f6  [1,18,4]   Successfully reassigned
 279 17611:08  000000011e1cb7f7  [1,18,4]   Recovered via rewrite in-place
 280 17611:08  000000011e1cb7fa  [1,17,1]   Successfully reassigned
 281 17611:08  000000011e1cb802  [1,18,4]   Successfully reassigned
 282 17611:08  000000011e1cb807  [1,18,4]   Successfully reassigned
 283 17611:08  000000011e1cb81e  [1,17,1]   Successfully reassigned
 284 17611:08  000000011e1cb81f  [1,18,4]   Successfully reassigned
 285 17611:08  000000011e1cb828  [1,18,1]   Successfully reassigned
 286 17611:08  000000011e1cb82d  [1,18,4]   Successfully reassigned
 287 17611:08  000000011e1d69fe  [1,17,1]   Successfully reassigned
 288 17611:08  000000011e98b183  [1,18,4]   Successfully reassigned
 289 17611:08  000000011e991932  [1,17,3]   Successfully reassigned
 290 17611:09  000000011ead3bc6  [1,18,4]   Recovered via rewrite in-place
 291 17611:09  000000011eadb9ee  [1,18,4]   Recovered via rewrite in-place
 292 17611:09  000000011eadb9ef  [1,18,4]   Recovered via rewrite in-place
 293 17611:09  000000011eb2d182  [1,18,4]   Successfully reassigned
 294 17611:09  000000011eb2d183  [1,18,4]   Recovered via rewrite in-place
 295 17611:09  000000011eb2d184  [1,18,4]   Successfully reassigned
 296 17611:09  000000011eb2e40e  [1,18,4]   Successfully reassigned
 297 17611:09  000000011eb2e40f  [1,18,4]   Recovered via rewrite in-place
 298 17611:09  000000011eb2e410  [1,18,4]   Recovered via rewrite in-place
 299 17611:09  000000011eb2e411  [1,18,4]   Recovered via rewrite in-place
 300 17611:09  000000011eb2e412  [1,18,4]   Recovered via rewrite in-place
 301 17611:09  000000011eb2e414  [1,18,4]   Successfully reassigned
 302 17611:09  000000011eb2e415  [1,18,4]   Successfully reassigned
 303 17611:09  000000011eb2e416  [1,18,4]   Recovered via rewrite in-place
 304 17611:09  000000011eb2e417  [1,18,4]   Recovered via rewrite in-place
 305 17611:09  000000011eba4f0a  [1,18,4]   Successfully reassigned
 306 17611:09  000000011eba4f0b  [1,18,4]   Recovered via rewrite in-place
 307 17611:09  000000011eba4f0c  [1,18,4]   Successfully reassigned
 308 17611:09  000000011eba4f0d  [1,18,4]   Successfully reassigned
 309 17611:09  000000011eba4f0e  [1,18,4]   Successfully reassigned
 310 17611:09  000000011eba4f0f  [1,18,4]   Successfully reassigned
 311 17611:09  000000011eba4f10  [1,18,4]   Successfully reassigned
 312 17611:09  000000011eba4f11  [1,18,4]   Recovered via rewrite in-place
 313 17611:09  000000011eba4f12  [1,18,4]   Successfully reassigned
 314 17611:09  000000011eba4f13  [1,18,4]   Successfully reassigned
 315 17611:09  000000011eba4f1f  [1,18,4]   Recovered via rewrite in-place
 316 17611:09  000000011eba4f21  [1,18,4]   Recovered via rewrite in-place
 317 17611:09  000000011eba4f22  [1,18,4]   Successfully reassigned
 318 17611:09  000000011eba4f23  [1,17,1]   Successfully reassigned
 319 17611:10  000000011f15c74c  [1,18,4]   Successfully reassigned
 320 17611:10  000000011f21e155  [1,18,4]   Successfully reassigned
 321 17611:10  000000011f34e65f  [1,18,4]   Successfully reassigned
 322 17611:10  000000011f34e660  [1,18,4]   Successfully reassigned
 323 17611:10  000000011f34e661  [1,18,4]   Successfully reassigned
 324 17611:10  000000011f34e662  [1,18,4]   Successfully reassigned
 325 17611:10  000000011f34e663  [1,17,1]   Successfully reassigned
 326 17611:10  000000011f34e66f  [1,18,4]   Recovered via rewrite in-place
 327 17611:10  000000011f34e673  [1,18,4]   Successfully reassigned
 328 17611:10  000000011f3bc77f  [1,18,4]   Successfully reassigned
 329 17611:10  000000011f3bc797  [1,18,4]   Successfully reassigned
 330 17611:10  000000011f3bc799  [1,18,4]   Recovered via rewrite in-place
 331 17611:10  000000011f3bc79b  [1,18,4]   Successfully reassigned
 332 17611:10  000000011f3bc79f  [1,18,4]   Successfully reassigned
 333 17611:10  000000011f4187a1  [1,18,4]   Successfully reassigned
 334 17611:10  000000011f4187a5  [1,18,4]   Recovered via rewrite in-place
 335 17611:10  000000011f4187c5  [1,18,4]   Recovered via rewrite in-place
 336 17611:10  000000011f4187c6  [1,18,4]   Successfully reassigned
 337 17611:10  000000011f6f7738  [1,18,4]   Successfully reassigned
 338 17611:10  000000011f6f7739  [1,18,4]   Successfully reassigned
 339 17611:10  000000011f6f773d  [1,18,4]   Successfully reassigned
 340 17611:10  000000011f6f773f  [1,18,4]   Successfully reassigned
 341 17611:10  000000011f74fa86  [1,18,4]   Successfully reassigned
 342 17611:10  000000011f75c079  [1,18,4]   Successfully reassigned
 343 17611:11  000000011f75c07a  [1,18,4]   Successfully reassigned
 344 17611:11  000000011f75c07b  [1,17,1]   Recovered via rewrite in-place
 345 17611:11  000000011f81c394  [1,18,4]   Recovered via rewrite in-place
 346 17611:11  000000011f81c395  [1,18,4]   Successfully reassigned
 347 17611:11  000000011f81c3a1  [1,17,1]   Successfully reassigned
 348 17611:11  000000011f81c3a2  [1,18,4]   Successfully reassigned
 349 17611:11  000000011f81c3a3  [1,17,1]   Successfully reassigned
 350 17611:11  000000011f81c3ab  [1,18,4]   Successfully reassigned
 351 17611:11  000000011faee8d4  [1,18,4]   Successfully reassigned
 352 17611:11  000000011fd9e905  [1,18,4]   Successfully reassigned
 353 17611:11  000000011fd9e906  [1,18,4]   Successfully reassigned
 354 17611:11  000000011ff32615  [1,18,4]   Successfully reassigned
 355 17611:11  000000011ff32618  [1,18,4]   Successfully reassigned
 356 17611:11  000000011ff32619  [1,18,4]   Recovered via rewrite in-place
 357 17611:11  000000011ff3261a  [1,18,4]   Recovered via rewrite in-place
 358 17611:11  000000011ff3261b  [1,18,4]   Recovered via rewrite in-place
 359 17611:11  000000011ff3261c  [1,18,4]   Recovered via rewrite in-place
 360 17611:11  000000011ff3262e  [1,18,4]   Successfully reassigned
 361 17611:11  000000011ff3262f  [1,18,4]   Successfully reassigned
 362 17611:11  000000011ff32630  [1,18,4]   Successfully reassigned
 363 17611:11  000000011ff32634  [1,18,4]   Successfully reassigned
 364 17611:11  000000011ff32636  [1,18,4]   Successfully reassigned
 365 17611:11  000000011ff32639  [1,18,4]   Recovered via rewrite in-place
 366 17611:11  000000011ff3263a  [1,18,4]   Successfully reassigned
 367 17611:12  00000001200c0543  [1,18,4]   Successfully reassigned
 368 17611:12  0000000120134d6d  [1,18,4]   Recovered via rewrite in-place
 369 17611:12  0000000120134d6e  [1,18,4]   Successfully reassigned
 370 17611:12  0000000120134d70  [1,18,4]   Recovered via rewrite in-place
 371 17611:12  00000001202c7aca  [1,18,4]   Successfully reassigned
 372 17611:13  0000000120b8494a  [1,17,1]   Successfully reassigned
 373 17611:13  0000000120b8624c  [1,18,4]   Successfully reassigned
 374 17611:13  00000001213ce9fb  [1,18,4]   Recovered via rewrite in-place
 375 17611:13  00000001213ce9fc  [1,18,4]   Successfully reassigned
 376 17611:13  000000012163925f  [1,18,4]   Successfully reassigned
 377 17611:13  0000000121639262  [1,17,1]   Successfully reassigned
 378 17611:13  00000001216b6df9  [1,18,4]   Successfully reassigned
 379 17611:13  00000001216bfc31  [1,18,4]   Recovered via rewrite in-place
 380 17611:14  0000000121706223  [1,18,4]   Successfully reassigned
 381 17611:15  00000001229c2b8d  [1,17,1]   Recovered via rewrite in-place
 382 17611:15  0000000122bdab7e  [1,17,1]   Successfully reassigned
 383 17611:15  0000000122bdab7f  [1,18,4]   Successfully reassigned
 384 17611:15  0000000122bdab80  [1,18,4]   Successfully reassigned
 385 17611:15  0000000122c37787  [1,18,4]   Successfully reassigned
 386 17611:15  0000000122c3778b  [1,18,4]   Successfully reassigned
 387 17611:15  0000000122c37790  [1,18,4]   Successfully reassigned
 388 17611:15  0000000122c37791  [1,18,4]   Successfully reassigned
 389 17611:15  0000000122c37792  [1,18,4]   Recovered via rewrite in-place
 390 17611:15  0000000122c37794  [1,18,4]   Successfully reassigned
 391 17611:15  0000000122c37795  [1,18,4]   Recovered via rewrite in-place
 392 17611:15  0000000122c37798  [1,18,4]   Successfully reassigned
 393 17611:15  0000000122c37799  [1,18,4]   Recovered via rewrite in-place
 394 17611:15  0000000122c3779a  [1,18,4]   Successfully reassigned
 395 17611:15  0000000122c3779b  [1,18,4]   Successfully reassigned
 396 17611:15  0000000122c3779e  [1,18,4]   Successfully reassigned
 397 17611:15  0000000122c377a0  [1,18,4]   Successfully reassigned
 398 17611:15  0000000122c377aa  [1,18,4]   Successfully reassigned
 399 17611:15  0000000122c377b5  [1,17,1]   Successfully reassigned
 400 17611:15  0000000122c377b6  [1,18,4]   Successfully reassigned
 401 17611:15  0000000122c377b7  [1,17,1]   Successfully reassigned
 402 17611:15  0000000122c377b9  [1,18,4]   Successfully reassigned
 403 17611:15  0000000122cbdf0c  [1,18,4]   Recovered via rewrite in-place
 404 17611:15  0000000122cbdf27  [1,18,4]   Successfully reassigned
 405 17611:15  0000000122cbdf29  [1,17,1]   Successfully reassigned
 406 17611:16  0000000122ea8e88  [1,18,4]   Successfully reassigned
 407 17611:16  00000001230c597e  [1,18,4]   Successfully reassigned
 408 17611:16  00000001231ef1fd  [1,18,4]   Successfully reassigned
 409 17611:16  000000012333b07c  [1,17,1]   Successfully reassigned
 410 17611:16  000000012333b07e  [1,18,4]   Successfully reassigned
 411 17611:16  000000012333b6a6  [1,18,4]   Successfully reassigned
 412 17611:16  000000012333b6a7  [1,18,4]   Successfully reassigned
 413 17611:16  000000012333b6ae  [1,18,4]   Successfully reassigned
 414 17611:16  00000001236830b1  [1,18,4]   Recovered via rewrite in-place
 415 17611:16  00000001236830b2  [1,18,4]   Successfully reassigned
 416 17611:16  0000000123752304  [1,18,4]   Successfully reassigned
 417 17611:17  000000012388c63f  [1,18,4]   Successfully reassigned
 418 17611:17  000000012388c643  [1,17,1]   Successfully reassigned
 419 17611:19  0000000125441819  [1,17,1]   Successfully reassigned
 420 17611:19  0000000125e7065c  [1,18,4]   Successfully reassigned
 421 17611:19  0000000125e7065e  [1,18,4]   Recovered via rewrite in-place
 422 17611:19  0000000125e7065f  [1,18,4]   Successfully reassigned
 423 17611:21  0000000127b713ae  [1,18,4]   Recovered via rewrite in-place
 424 17611:22  00000001281141ac  [1,18,4]   Successfully reassigned
 425 17611:22  00000001281141ae  [1,18,4]   Successfully reassigned
 426 17611:22  00000001281e247b  [1,18,4]   Successfully reassigned
 427 17611:22  00000001281e247c  [1,17,1]   Successfully reassigned
 428 17611:23  0000000128973330  [1,18,4]   Successfully reassigned
 429 17611:23  0000000128d75e9d  [1,18,4]   Successfully reassigned
 430 17611:23  0000000128d75ea0  [1,18,4]   Recovered via rewrite in-place
 431 17611:36  00000001346bab9f  [1,17,1]   Successfully reassigned
 432 17611:43  0000000139e28cf7  [1,18,4]   Successfully reassigned
 433 17612:00  0000000147f5240c  [1,18,4]   Successfully reassigned
 434 17617:36  000000007643ce66  [1,18,4]   Successfully reassigned
 435 17618:11  00000000a0c8f93e  [1,18,4]   Successfully reassigned
 436 17618:11  00000000a0ca12ae  [1,18,4]   Successfully reassigned
 437 17618:11  00000000a0da69ad  [1,18,4]   Successfully reassigned
 438 17618:11  00000000a0dab07d  [1,18,4]   Successfully reassigned
 439 17618:12  00000000a0f38ccd  [1,17,1]   Successfully reassigned
 440 17618:12  00000000a0f46bbb  [1,18,4]   Successfully reassigned
 441 17618:12  00000000a0f48322  [1,18,4]   Successfully reassigned
 442 17618:12  00000000a122526e  [1,18,4]   Successfully reassigned
 443 17618:12  00000000a12c85b9  [1,18,4]   Successfully reassigned
 444 17618:12  00000000a14526e5  [1,18,4]   Successfully reassigned
 445 17618:12  00000000a14578ce  [1,18,4]   Successfully reassigned
 446 17618:12  00000000a1464055  [1,18,4]   Successfully reassigned
 447 17618:12  00000000a15039c1  [1,18,4]   Successfully reassigned
 448 17618:12  00000000a156bb11  [1,18,4]   Successfully reassigned
 449 17619:58  0000000113c42756  [1,18,4]   Successfully reassigned
 450 17620:50  0000000141dfadec  [1,17,1]   Successfully reassigned
 451 17620:51  00000001420d5f78  [1,18,4]   Successfully reassigned
 452 17620:51  00000001420d5f9a  [1,18,4]   Recovered via rewrite in-place
 453 17620:51  00000001420d5f9b  [1,18,4]   Successfully reassigned
 454 17620:51  000000014218ca5e  [1,17,1]   Successfully reassigned
 455 17620:51  00000001421f6130  [1,18,4]   Successfully reassigned
 456 17620:54  00000001445ccd1d  [1,18,4]   Recovered via rewrite in-place
 457 17620:54  000000014468274f  [1,18,4]   Recovered via rewrite in-place
 458 17620:54  0000000144bd2dee  [1,18,4]   Successfully reassigned
 459 17620:54  0000000144bd2df0  [1,17,1]   Successfully reassigned
 460 17620:54  0000000144bdb700  [1,18,4]   Successfully reassigned
 461 17620:54  0000000144bdb714  [1,17,1]   Successfully reassigned
 462 17620:54  0000000144bdb72b  [1,18,4]   Successfully reassigned
 463 17621:12  00000001522f56f9  [1,17,1]   Successfully reassigned
 464 17621:12  00000001522f56fc  [1,18,4]   Successfully reassigned
 465 17621:12  00000001522f56fd  [1,17,1]   Recovered via rewrite in-place
 466 17621:12  00000001522f56ff  [1,18,4]   Successfully reassigned
 467 17621:12  00000001522f5700  [1,18,4]   Recovered via rewrite in-place
 468 17621:12  00000001522f5709  [1,18,4]   Recovered via rewrite in-place
 469 17621:12  00000001522f570a  [1,18,4]   Recovered via rewrite in-place
 470 17621:12  00000001522f570b  [1,18,4]   Recovered via rewrite in-place
 471 17621:12  00000001522f5712  [1,18,4]   Recovered via rewrite in-place
 472 17621:12  00000001522f5728  [1,18,4]   Successfully reassigned
 473 17621:12  00000001522f5750  [1,18,4]   Recovered via rewrite in-place
 474 17621:12  0000000152455afd  [1,18,4]   Successfully reassigned
 475 17621:12  0000000152455afe  [1,18,4]   Successfully reassigned
 476 17621:12  0000000152455b07  [1,18,4]   Successfully reassigned
 477 17621:12  0000000152455b11  [1,18,4]   Successfully reassigned
 478 17621:12  000000015245fdcc  [1,17,1]   Successfully reassigned
 479 17621:12  000000015245fdd0  [1,18,4]   Successfully reassigned
 480 17621:12  00000001524ba081  [1,18,4]   Successfully reassigned
 481 17621:12  00000001524bc2ec  [1,18,4]   Successfully reassigned
 482 17621:12  00000001524be2a1  [1,17,1]   Successfully reassigned
 483 17621:12  00000001524be2a2  [1,18,4]   Successfully reassigned
 484 17621:12  00000001524be2a6  [1,18,4]   Recovered via rewrite in-place
 485 17621:12  00000001524be2a7  [1,18,4]   Successfully reassigned
 486 17621:12  00000001524be2aa  [1,18,4]   Successfully reassigned
 487 17621:12  00000001524be2ab  [1,18,4]   Recovered via rewrite in-place
 488 17621:12  00000001524be2ad  [1,18,4]   Successfully reassigned
 489 17621:12  00000001524be2ae  [1,18,4]   Successfully reassigned
 490 17621:12  00000001524be2af  [1,18,4]   Recovered via rewrite in-place
 491 17621:12  00000001524be2b2  [1,18,4]   Successfully reassigned
 492 17621:13  00000001524be2d6  [1,18,4]   Successfully reassigned
 493 17621:13  00000001529d4f25  [1,18,4]   Successfully reassigned
 494 17621:13  00000001529d4f2b  [1,18,4]   Successfully reassigned
 495 17621:13  00000001529d4f31  [1,18,4]   Successfully reassigned
 496 17621:13  00000001529d4f32  [1,18,4]   Recovered via rewrite in-place
 497 17621:13  00000001529d4f33  [1,18,4]   Recovered via rewrite in-place
 498 17621:13  00000001529d4f36  [1,18,4]   Recovered via rewrite in-place
 499 17621:13  00000001529d4f37  [1,18,4]   Successfully reassigned
 500 17621:13  00000001529d4f3b  [1,18,4]   Successfully reassigned
 501 17621:13  00000001529d4f3c  [1,18,4]   Recovered via rewrite in-place
 502 17621:13  00000001529dea3e  [1,18,4]   Successfully reassigned
 503 17621:13  00000001529dea42  [1,18,4]   Successfully reassigned
 504 17621:13  0000000152a45676  [1,18,4]   Successfully reassigned
 505 17621:13  0000000152a45679  [1,18,4]   Recovered via rewrite in-place
 506 17621:13  0000000152a4567a  [1,18,4]   Successfully reassigned
 507 17621:13  0000000152a82ada  [1,18,4]   Successfully reassigned
 508 17621:13  0000000152a82adc  [1,17,1]   Successfully reassigned
 509 17621:13  0000000152a82ae3  [1,18,4]   Successfully reassigned
 510 17621:13  0000000152a82ae5  [1,18,4]   Successfully reassigned
 511 17621:13  0000000152a82ae6  [1,17,1]   Successfully reassigned
 512 17621:13  0000000152a82aee  [1,18,4]   Successfully reassigned
 513 17621:13  0000000152a82af0  [1,17,1]   Successfully reassigned
 514 17621:13  0000000152a82af7  [1,18,4]   Successfully reassigned
 515 17621:14  0000000152daf41d  [1,18,4]   Successfully reassigned
 516 17621:14  0000000152daf421  [1,17,1]   Successfully reassigned
 517 17621:14  0000000152df2e9d  [1,18,4]   Successfully reassigned
 518 17621:14  0000000152dfc0f8  [1,17,1]   Successfully reassigned
 519 17621:14  0000000152ea06e6  [1,17,1]   Successfully reassigned
 520 17621:14  0000000152ea06ec  [1,18,4]   Successfully reassigned
 521 17621:14  0000000152f4ec7d  [1,18,4]   Recovered via rewrite in-place
 522 17621:14  0000000152f4ec87  [1,18,4]   Successfully reassigned
 523 17621:14  0000000152f4ec88  [1,18,4]   Recovered via rewrite in-place
 524 17621:14  0000000152f4ec8a  [1,18,4]   Recovered via rewrite in-place
 525 17621:14  0000000152f4ec8e  [1,18,4]   Recovered via rewrite in-place
 526 17621:14  0000000152f4ec8f  [1,18,4]   Recovered via rewrite in-place
 527 17621:14  0000000152f4ec90  [1,17,1]   Successfully reassigned
 528 17621:14  0000000152f4ec91  [1,18,4]   Recovered via rewrite in-place
 529 17621:14  0000000152fbb493  [1,18,4]   Successfully reassigned
 530 17621:14  0000000152fbb494  [1,17,1]   Successfully reassigned
 531 17621:14  0000000152fbb497  [1,18,4]   Successfully reassigned
 532 17621:14  0000000152fbb499  [1,18,4]   Successfully reassigned
 533 17621:14  0000000152fbb49a  [1,18,4]   Recovered via rewrite in-place
 534 17621:14  0000000152fbb49c  [1,18,4]   Recovered via rewrite in-place
 535 17621:14  0000000152fbb49d  [1,18,4]   Successfully reassigned
 536 17621:14  0000000152fbb49e  [1,18,4]   Successfully reassigned
 537 17621:14  0000000152fc3b12  [1,18,4]   Recovered via rewrite in-place
 538 17621:14  0000000152fc3b1b  [1,17,1]   Successfully reassigned
 539 17621:14  0000000152fc3b1d  [1,18,4]   Successfully reassigned
 540 17621:14  0000000152fc3b21  [1,18,4]   Recovered via rewrite in-place
 541 17621:15  0000000153064c55  [1,18,4]   Successfully reassigned
 542 17621:15  000000015306bc10  [1,18,4]   Successfully reassigned
 543 17621:15  000000015306bc14  [1,18,4]   Recovered via rewrite in-place
 544 17621:15  000000015306bc15  [1,18,4]   Successfully reassigned
 545 17621:15  0000000153072859  [1,18,4]   Successfully reassigned
 546 17621:15  00000001530b19ed  [1,18,4]   Recovered via rewrite in-place
 547 17621:15  00000001530b19ef  [1,18,4]   Recovered via rewrite in-place
 548 17621:15  00000001530b282b  [1,17,1]   Successfully reassigned
 549 17621:15  00000001530b282e  [1,18,4]   Successfully reassigned
 550 17621:15  00000001531121bd  [1,18,4]   Recovered via rewrite in-place
 551 17621:15  00000001531121bf  [1,18,4]   Successfully reassigned
 552 17621:15  00000001531207b1  [1,18,4]   Successfully reassigned
 553 17621:15  00000001531207b2  [1,17,1]   Recovered via rewrite in-place
 554 17621:15  000000015315c939  [1,18,4]   Recovered via rewrite in-place
 555 17621:15  000000015315c93a  [1,18,4]   Recovered via rewrite in-place
 556 17621:15  000000015315c93b  [1,18,4]   Successfully reassigned
 557 17621:15  000000015315c93c  [1,18,4]   Recovered via rewrite in-place
 558 17621:15  000000015315c93d  [1,18,4]   Recovered via rewrite in-place
 559 17621:15  00000001531616d3  [1,18,4]   Successfully reassigned
 560 17621:15  00000001531616d4  [1,18,4]   Successfully reassigned
 561 17621:15  00000001531616d8  [1,18,4]   Recovered via rewrite in-place
 562 17621:15  00000001531616d9  [1,18,4]   Recovered via rewrite in-place
 563 17621:15  00000001531616da  [1,18,4]   Recovered via rewrite in-place
 564 17621:15  00000001531616db  [1,18,4]   Recovered via rewrite in-place
 565 17621:15  00000001531616e2  [1,18,4]   Recovered via rewrite in-place
 566 17621:15  00000001531616e3  [1,18,4]   Successfully reassigned
 567 17621:15  00000001531616e6  [1,18,4]   Recovered via rewrite in-place
 568 17621:15  00000001531616e7  [1,18,4]   Successfully reassigned
 569 17621:15  000000015316170e  [1,18,4]   Recovered via rewrite in-place
 570 17621:15  0000000153161710  [1,18,4]   Recovered via rewrite in-place
 571 17621:15  0000000153161719  [1,17,1]   Successfully reassigned
 572 17621:15  000000015316173b  [1,18,4]   Recovered via rewrite in-place
 573 17621:15  0000000153165bd2  [1,18,4]   Recovered via rewrite in-place
 574 17621:15  0000000153165bd3  [1,18,4]   Successfully reassigned
 575 17621:15  000000015316a61f  [1,18,4]   Successfully reassigned
 576 17621:15  000000015316a623  [1,18,4]   Recovered via rewrite in-place
 577 17621:15  000000015316a624  [1,18,4]   Successfully reassigned
 578 17621:15  000000015316a625  [1,18,4]   Successfully reassigned
 579 17621:15  000000015316a626  [1,18,4]   Successfully reassigned
 580 17621:15  000000015316a628  [1,18,4]   Successfully reassigned
 581 17621:15  000000015316a629  [1,18,4]   Successfully reassigned
 582 17621:15  000000015316a62a  [1,18,4]   Successfully reassigned
 583 17621:15  000000015316a62b  [1,18,4]   Successfully reassigned
 584 17621:15  000000015316a62d  [1,17,1]   Successfully reassigned
 585 17621:16  000000015320eb0d  [1,18,4]   Recovered via rewrite in-place
 586 17621:16  000000015320eb2c  [1,18,4]   Successfully reassigned
 587 17621:16  00000001532111fd  [1,17,1]   Successfully reassigned
 588 17621:16  00000001532119b4  [1,18,4]   Recovered via rewrite in-place
 589 17621:16  00000001532119b7  [1,18,4]   Recovered via rewrite in-place
 590 17621:16  00000001532119b9  [1,18,4]   Successfully reassigned
 591 17621:16  00000001532119ba  [1,18,4]   Recovered via rewrite in-place
 592 17621:16  00000001532119bb  [1,18,4]   Successfully reassigned
 593 17621:16  00000001532119bc  [1,17,1]   Successfully reassigned
 594 17621:16  00000001532119c2  [1,18,4]   Successfully reassigned
 595 17621:16  00000001532119c9  [1,18,4]   Successfully reassigned
 596 17621:16  00000001532119e7  [1,18,4]   Successfully reassigned
 597 17621:16  00000001532119e8  [1,18,4]   Successfully reassigned
 598 17621:16  00000001532144dd  [1,18,4]   Successfully reassigned
 599 17621:16  00000001532144de  [1,18,4]   Recovered via rewrite in-place
 600 17621:16  00000001532144df  [1,18,4]   Recovered via rewrite in-place
 601 17621:16  00000001532144e1  [1,18,4]   Successfully reassigned
 602 17621:16  00000001532144e2  [1,18,4]   Successfully reassigned
 603 17621:16  00000001532144e3  [1,18,4]   Recovered via rewrite in-place
 604 17621:16  00000001532144e4  [1,18,4]   Successfully reassigned
 605 17621:16  00000001532144e5  [1,18,4]   Successfully reassigned
 606 17621:16  00000001532144ec  [1,17,1]   Recovered via rewrite in-place
 607 17621:16  00000001532144ed  [1,18,4]   Recovered via rewrite in-place
 608 17621:16  00000001532144ee  [1,18,4]   Recovered via rewrite in-place
 609 17621:16  00000001532144ef  [1,18,4]   Successfully reassigned
 610 17621:16  00000001532144f0  [1,18,4]   Successfully reassigned
 611 17621:16  00000001532144f1  [1,18,4]   Recovered via rewrite in-place
 612 17621:16  00000001532144f2  [1,18,4]   Recovered via rewrite in-place
 613 17621:16  00000001532144f3  [1,18,4]   Successfully reassigned
 614 17621:16  00000001532144f4  [1,18,4]   Successfully reassigned
 615 17621:16  00000001532144f5  [1,18,4]   Recovered via rewrite in-place
 616 17621:16  00000001532144f6  [1,18,4]   Recovered via rewrite in-place
 617 17621:16  00000001532144f7  [1,18,4]   Successfully reassigned
 618 17621:16  00000001532144f8  [1,18,4]   Recovered via rewrite in-place
 619 17621:16  00000001532144ff  [1,18,4]   Successfully reassigned
 620 17621:16  000000015321450a  [1,18,4]   Successfully reassigned
 621 17621:16  00000001532c9a7b  [1,18,4]   Successfully reassigned
 622 17621:16  00000001532c9a9d  [1,18,4]   Successfully reassigned
 623 17621:16  00000001532c9a9e  [1,18,4]   Recovered via rewrite in-place
 624 17621:16  00000001532c9a9f  [1,18,4]   Successfully reassigned
 625 17621:16  00000001532c9aa0  [1,18,4]   Recovered via rewrite in-place
 626 17621:16  00000001532c9aa1  [1,18,4]   Recovered via rewrite in-place
 627 17621:16  00000001532c9aa2  [1,18,4]   Recovered via rewrite in-place
 628 17621:16  00000001532c9aa3  [1,18,4]   Recovered via rewrite in-place
 629 17621:16  000000015331fe86  [1,18,4]   Successfully reassigned
 630 17621:16  000000015331fe87  [1,18,4]   Successfully reassigned
 631 17621:16  000000015331fe89  [1,18,4]   Recovered via rewrite in-place
 632 17621:16  000000015331fe8a  [1,18,4]   Recovered via rewrite in-place
 633 17621:17  0000000153324c61  [1,18,4]   Successfully reassigned
 634 17621:17  0000000153324c63  [1,18,4]   Recovered via rewrite in-place
 635 17621:17  0000000153324c64  [1,18,4]   Recovered via rewrite in-place
 636 17621:17  0000000153324c65  [1,18,4]   Successfully reassigned
 637 17621:17  0000000153324c67  [1,18,4]   Successfully reassigned
 638 17621:17  0000000153324c68  [1,18,4]   Recovered via rewrite in-place
 639 17621:17  0000000153324c69  [1,18,4]   Successfully reassigned
 640 17621:17  0000000153324c6c  [1,18,4]   Recovered via rewrite in-place
 641 17621:17  0000000153324c71  [1,18,4]   Successfully reassigned
 642 17621:17  0000000153324c9b  [1,18,4]   Successfully reassigned
 643 17621:17  000000015332bbb8  [1,18,4]   Successfully reassigned
 644 17621:17  000000015332bbb9  [1,18,4]   Successfully reassigned
 645 17621:17  000000015332bbba  [1,17,1]   Recovered via rewrite in-place
 646 17621:17  000000015332bbbd  [1,18,4]   Successfully reassigned
 647 17621:17  000000015332bbbe  [1,18,4]   Successfully reassigned
 648 17621:17  000000015332bbbf  [1,18,4]   Successfully reassigned
 649 17621:17  000000015332bbc0  [1,18,4]   Recovered via rewrite in-place
 650 17621:17  000000015369042f  [1,18,4]   Successfully reassigned
 651 17621:17  0000000153697887  [1,18,4]   Recovered via rewrite in-place
 652 17621:17  00000001536db075  [1,18,4]   Recovered via rewrite in-place
 653 17621:17  00000001536db076  [1,18,4]   Recovered via rewrite in-place
 654 17621:17  00000001536db078  [1,18,4]   Successfully reassigned
 655 17621:17  00000001536e53a9  [1,18,4]   Recovered via rewrite in-place
 656 17621:17  00000001536e53ac  [1,18,4]   Successfully reassigned
 657 17621:17  00000001536e53b2  [1,18,4]   Recovered via rewrite in-place
 658 17621:17  000000015373b81c  [1,17,1]   Recovered via rewrite in-place
 659 17621:17  000000015373b81d  [1,18,4]   Recovered via rewrite in-place
 660 17621:17  000000015373b81e  [1,18,4]   Recovered via rewrite in-place
 661 17621:17  0000000153747a0d  [1,18,4]   Recovered via rewrite in-place
 662 17621:17  0000000153747a13  [1,18,4]   Successfully reassigned
 663 17621:17  0000000153747a14  [1,18,4]   Successfully reassigned
 664 17621:17  0000000153747a15  [1,18,4]   Successfully reassigned
 665 17621:17  0000000153747a23  [1,18,4]   Successfully reassigned
 666 17621:18  0000000153747a24  [1,18,4]   Recovered via rewrite in-place
 667 17621:20  000000015568a99f  [1,18,4]   Successfully reassigned
 668 17621:21  0000000155b46fd8  [1,18,4]   Successfully reassigned
 669 17621:21  0000000155b9d1a8  [1,18,4]   Successfully reassigned
 670 17621:22  00000001567c059f  [1,18,4]   Recovered via rewrite in-place
 671 17621:22  00000001567c05a0  [1,18,4]   Successfully reassigned
 >>>> log truncated, fetched 16124 of 49244 available bytes

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 0
  number of phys = 1
  phy identifier = 0
    attached device type: SAS or SATA device
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=1 stp=1 smp=1
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000c50093a8f815
    attached SAS address = 0x500605b006a3bfe5
    attached phy identifier = 3
    Invalid DWORD count = 90
    Running disparity error count = 77
    Loss of DWORD synchronization = 87
    Phy reset problem = 3
    Phy event descriptors:
     Invalid word count: 90
     Running disparity error count: 77
     Loss of dword synchronization count: 87
     Phy reset problem count: 3
relative target port id = 2
  generation code = 0
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000c50093a8f816
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 4
    Phy reset problem = 1
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 4
     Phy reset problem count: 1

 

[mr44er]

for HGST He10 is 1008 hours. Did they found that 168 hours is a bit of overkill?

Possible. Or maybe it started to show performance issues on newer firmware, also I don't have experience with newer drives. This He means helium?

It depends on the vendor realization. I've had bad experience with Seagate. Maybe that's why storage vendors such as Netapp and EMC

Yes, afaik many models had problems on the first firmware during 2009-2011...fallouts, read errors and up to early self destruction via time bomb bugs. So every vendor downstream needed to push that to the clients servers in time.

Spoiler: smart seagate

smartctl -x -q noserial /dev/da18
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.2-RELEASE amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               IBM-ESXS
Product:              ST2000NM0001
Revision:             BC4B
Compliance:           SPC-4
User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Device type:          disk
Transport protocol:   SAS (SPL-4)
Local Time is:        Fri Jun  2 12:26:11 2023 CEST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Disabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     43 C
Drive Trip Temperature:        65 C

Elements in grown defect list: 0

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   4185355116        0         0  4185355116          0     298446.602           0
write:         0        0         0         0          0     103070.613           0
verify: 2481793015        0         0  2481793015          0       2159.756           0

Non-medium error count:        7

No Self-tests have been logged

Background scan results log
  Status: waiting until BMS interval timer expires
    Accumulated power on time, hours:minutes 53923:09 [3235389 minutes]
    Number of background scans performed: 720,  scan progress: 0.00%
    Number of background medium scans performed: 720

   #  when        lba(hex)    [sk,asc,ascq]    reassign_status
   1 4672:53  00000000957bbd57  [1,17,1]   Recovered via rewrite in-place
   2 5445:49  00000000a0b23c00  [1,17,1]   Recovered via rewrite in-place
   3 6011:01  00000000983c5000  [1,17,1]   Recovered via rewrite in-place
   4 6117:03  00000000c8a9b937  [1,17,1]   Recovered via rewrite in-place
   5 6451:25  000000009178ce00  [1,17,1]   Recovered via rewrite in-place
   6 7183:54  00000000117e6450  [1,17,1]   Recovered via rewrite in-place
   7 7183:54  00000000117e6451  [1,17,1]   Recovered via rewrite in-place
   8 7364:51  00000000e5d55e76  [1,17,1]   Recovered via rewrite in-place
   9 7678:31  0000000005c55ab8  [1,18,7]   Recovered via rewrite in-place
  10 7744:59  000000006723e84a  [1,17,1]   Recovered via rewrite in-place
  11 7993:36  0000000000001806  [1,18,7]   Recovered via rewrite in-place
  12 7995:06  000000000000185a  [1,18,7]   Recovered via rewrite in-place
  13 8222:43  0000000017d6d3e8  [1,17,1]   Recovered via rewrite in-place
  14 9961:46  00000000abbcc800  [1,17,1]   Recovered via rewrite in-place
  15 11168:58  0000000002fa7ee9  [1,17,1]   Recovered via rewrite in-place
  16 11525:02  00000000c10f8200  [1,17,1]   Recovered via rewrite in-place
  17 12622:26  00000000000014fa  [1,18,7]   Recovered via rewrite in-place
  18 12624:30  0000000000000df8  [1,18,7]   Recovered via rewrite in-place
  19 12634:49  0000000000001f5a  [1,18,7]   Recovered via rewrite in-place
  20 12679:58  000000000ea9ceb8  [1,17,1]   Recovered via rewrite in-place
  21 12733:33  000000007a9a8000  [1,17,1]   Recovered via rewrite in-place
  22 13059:11  0000000076ef122c  [1,17,1]   Recovered via rewrite in-place
  23 13250:42  000000007d58c000  [1,17,1]   Recovered via rewrite in-place
  24 13346:26  00000000c93891e9  [1,17,1]   Recovered via rewrite in-place
  25 13485:37  0000000070a9ab95  [1,17,1]   Recovered via rewrite in-place
  26 13730:44  0000000076189400  [1,17,1]   Recovered via rewrite in-place
  27 14133:03  00000000a88d8c00  [1,17,1]   Recovered via rewrite in-place
  28 14337:26  0000000000001a20  [1,18,7]   Recovered via rewrite in-place
  29 14490:56  0000000000001802  [1,18,7]   Recovered via rewrite in-place
  30 15191:59  00000000b5596a00  [1,17,1]   Recovered via rewrite in-place
  31 15341:56  000000003b22f8bc  [1,17,1]   Recovered via rewrite in-place
  32 15904:30  0000000079b00e00  [1,17,1]   Recovered via rewrite in-place
  33 16612:02  00000000a052ea00  [1,17,1]   Recovered via rewrite in-place
  34 17741:04  00000000000020da  [1,18,7]   Recovered via rewrite in-place
  35 17773:24  0000000000001501  [1,18,7]   Recovered via rewrite in-place
  36 18713:56  00000000450e17e9  [1,17,1]   Recovered via rewrite in-place
  37 19206:24  0000000000001f75  [1,18,7]   Recovered via rewrite in-place
  38 19511:42  0000000000001df8  [1,18,7]   Recovered via rewrite in-place
  39 20191:35  000000000679f54a  [1,17,1]   Recovered via rewrite in-place
  40 20218:25  0000000000001d75  [1,18,7]   Recovered via rewrite in-place
  41 20289:47  0000000000001c03  [1,18,7]   Recovered via rewrite in-place
  42 20289:47  0000000000001c54  [1,18,7]   Recovered via rewrite in-place
  43 20420:15  00000000bdcc0f06  [1,17,1]   Recovered via rewrite in-place
  44 20821:19  0000000000000cf1  [1,18,7]   Recovered via rewrite in-place
  45 20981:47  00000000000014fa  [1,18,7]   Recovered via rewrite in-place
  46 20991:24  0000000039819800  [1,17,1]   Recovered via rewrite in-place
  47 21335:18  0000000000001863  [1,18,7]   Recovered via rewrite in-place
  48 21449:01  00000000957bbd59  [1,17,1]   Recovered via rewrite in-place
  49 21705:38  0000000000002006  [1,18,7]   Recovered via rewrite in-place
  50 21803:00  0000000000001a70  [1,18,7]   Recovered via rewrite in-place
  51 22025:57  00000000c8a55600  [1,17,1]   Recovered via rewrite in-place
  52 22049:13  000000004682bfa9  [1,17,1]   Recovered via rewrite in-place
  53 22103:00  000000003de42400  [1,17,1]   Recovered via rewrite in-place
  54 22774:14  000000003aaf6a90  [1,17,1]   Recovered via rewrite in-place
  55 22927:31  00000000c3a581e6  [1,17,1]   Recovered via rewrite in-place
  56 23229:12  00000000000019a6  [1,18,7]   Recovered via rewrite in-place
  57 25627:31  0000000000001ae5  [1,18,7]   Recovered via rewrite in-place
  58 25640:25  0000000000001f6e  [1,18,7]   Recovered via rewrite in-place
  59 25713:10  000000000000193b  [1,18,7]   Recovered via rewrite in-place
  60 26524:41  0000000000001506  [1,18,7]   Recovered via rewrite in-place
  61 26926:02  0000000093a2e9ea  [1,17,1]   Recovered via rewrite in-place
  62 26983:52  0000000000001b80  [1,18,7]   Recovered via rewrite in-place
  63 27453:51  000000005d213a19  [1,17,1]   Recovered via rewrite in-place
  64 28663:46  0000000000001805  [1,18,7]   Recovered via rewrite in-place
  65 29632:14  00000000000021b6  [1,18,7]   Recovered via rewrite in-place
  66 30051:54  00000000af2bfb9c  [1,17,1]   Recovered via rewrite in-place
  67 30383:10  00000000c6095a00  [1,17,1]   Recovered via rewrite in-place
  68 31181:50  0000000000001f5f  [1,18,7]   Recovered via rewrite in-place
  69 31747:48  00000000000014ff  [1,18,7]   Recovered via rewrite in-place
  70 32133:42  0000000000001d0d  [1,18,7]   Recovered via rewrite in-place
  71 32214:25  00000000000019f9  [1,18,7]   Recovered via rewrite in-place
  72 32412:05  0000000000001f0d  [1,18,7]   Recovered via rewrite in-place
  73 35202:45  00000000000018e5  [1,18,7]   Recovered via rewrite in-place
  74 35730:57  000000007a2a87a2  [1,17,1]   Recovered via rewrite in-place
  75 35962:09  00000000000017ff  [1,18,7]   Recovered via rewrite in-place
  76 36229:35  0000000000001c6f  [1,18,7]   Recovered via rewrite in-place
  77 36598:08  00000000a0b8fbb7  [1,17,1]   Recovered via rewrite in-place
  78 36942:24  000000000000185e  [1,18,7]   Recovered via rewrite in-place
  79 36964:20  0000000000000c07  [1,18,7]   Recovered via rewrite in-place
  80 37415:03  0000000000001dba  [1,18,7]   Recovered via rewrite in-place
  81 37694:04  0000000000001c16  [1,18,7]   Recovered via rewrite in-place
  82 38252:00  0000000000000d27  [1,18,7]   Recovered via rewrite in-place
  83 38252:00  00000000000019a4  [1,18,7]   Recovered via rewrite in-place
  84 38259:38  00000000000020f9  [1,18,7]   Recovered via rewrite in-place
  85 38260:50  0000000000001519  [1,18,7]   Recovered via rewrite in-place
  86 38312:00  00000000b168b1c5  [1,17,1]   Recovered via rewrite in-place
  87 38618:20  0000000000001aae  [1,18,7]   Recovered via rewrite in-place
  88 38941:15  0000000000001f65  [1,18,7]   Recovered via rewrite in-place
  89 39150:12  0000000000001a53  [1,18,7]   Recovered via rewrite in-place
  90 39492:24  0000000000001805  [1,18,7]   Recovered via rewrite in-place
  91 39956:50  000000000000185a  [1,18,7]   Recovered via rewrite in-place
  92 40007:11  00000000000019f6  [1,18,7]   Recovered via rewrite in-place
  93 40249:26  000000000d9aaf0d  [1,17,1]   Recovered via rewrite in-place
  94 40427:21  000000000000208b  [1,18,7]   Recovered via rewrite in-place
  95 42125:52  0000000000001e4a  [1,18,7]   Recovered via rewrite in-place
  96 42155:34  00000000a8e96b58  [1,17,1]   Recovered via rewrite in-place
  97 42692:20  00000000b6aca497  [1,17,1]   Recovered via rewrite in-place
  98 42909:34  00000000c8acf32a  [1,17,1]   Recovered via rewrite in-place
  99 42985:17  00000000a1d03e00  [1,17,1]   Recovered via rewrite in-place
100 43219:39  00000000000014fd  [1,18,7]   Recovered via rewrite in-place
101 43223:19  0000000000001f64  [1,18,7]   Recovered via rewrite in-place
102 43656:23  00000000bdfc5000  [1,17,1]   Recovered via rewrite in-place
103 43804:03  00000000abc02f75  [1,17,1]   Recovered via rewrite in-place
104 43914:39  0000000000001ea6  [1,18,7]   Recovered via rewrite in-place
105 44153:51  0000000000001b46  [1,18,7]   Recovered via rewrite in-place
106 44438:31  000000007fd8b71e  [1,17,1]   Recovered via rewrite in-place
107 44580:56  00000000c85eb367  [1,17,1]   Recovered via rewrite in-place
108 44798:20  0000000000005f95  [1,18,7]   Recovered via rewrite in-place
109 45031:33  0000000000001803  [1,18,7]   Recovered via rewrite in-place
110 45321:16  000000009b0c6f5d  [1,17,1]   Recovered via rewrite in-place
111 46690:17  00000000787769a7  [1,17,1]   Recovered via rewrite in-place
112 46690:31  0000000079d170d2  [1,17,1]   Recovered via rewrite in-place
113 52577:35  0000000000000b78  [1,18,7]   Recovered via rewrite in-place
114 53077:52  0000000000000be2  [1,18,7]   Recovered via rewrite in-place
115 53867:19  000000000db747cb  [1,17,1]   Recovered via rewrite in-place


Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 0
  number of phys = 1
  phy identifier = 0
    attached device type: expander device
    attached reason: loss of dword synchronization
    reason: power on
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=0 stp=0 smp=1
    attached target port: ssp=0 stp=0 smp=1
    attached phy identifier = 17
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization count = 0
    Phy reset problem count = 0
relative target port id = 2
  generation code = 0
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; 1.5 Gbps
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization count = 0
    Phy reset problem count = 0

 

[twin_savage]

Can't remember which video where Level1 Wendell says "the only filesystem I trust", referring to ZFS. Eliciting a response of great astonishment "Really??" from his guest. Yes, really. I wish Arch would support it out of the box, as root file system, with some optimizations pre-applied, booted through the excellent zfsbootmenu.

There are also some legitimate reasons not to trust ZFS. The fragmentation issue ZFS suffers coupled with the variable sized blocks that require reference to block pointer trees that are often fragmented themselves will cause hdd heads to thrash during resilver if the pool has been used for any reasonable amount of time and greatly increases the chance of additional drive failures compared to "traditional" raid.

 

[Fritz]

Are there any other file systems that address bit rot?

 

[twin_savage]

Are there any other file systems that address bit rot?

ReFS with data integrity streams enabled (not just metadata streams as is the default) is the only other file system that I would say does this well that isn't a work in progress (BTRFS).

IMO bitrot shouldn't be handled at the file system level to begin with, it should be done below the file system at the block level.
A crude analogy to handling bitrot at the filesystem level would be if OS/kernel developers tried to implement RAM ECC in the kernel with non-ECC RAM hardware by check summing memory values and storing redundant data in different areas of the RAM instead of just using hardware ECC RAM (the bitrot block level handling in this analogy).

 

[Whaaat]

storing redundant data in different areas of the RAM instead of just using hardware ECC RAM

ECC is not a replacement for page mirroring, ranks sparing, page offlining, memory scrambling and patrol scrubbing)) It's up to user to decide how much protection is enough. Servers have a little bit more to offer than ECC alone for memory RAS.

 

[twin_savage]

ECC is not a replacement for page mirroring, ranks sparing, page offlining, memory scrambling and patrol scrubbing)) It's up to user to decide how much protection is enough. Servers have a little bit more to offer than ECC alone for memory RAS.

This is very true, these functions could be included in that analogy to drive it home further; none of these RAS features belong in the kernel, they should be accomplished at a lower level.
I suspect the memory RAS features were initially implemented at a lower level in hardware for performance reasons as opposed to clean system design though, which is coincidentally the same reason data integrity was initially handled at the block level in hardware rather than kernel; but as CPUs got faster and storage didn't, Sun had the idea to move it up to the kernel since it was within the realm of possibility.

 

[oneplane]

IMO bitrot shouldn't be handled at the file system level to begin with, it should be done below the file system at the block level.

It shouldn't be done at one level either way, and luckily it isn't. Even spinning rust for example already uses ECC. But it simply isn't enough, both because it often still results in data loss (what do you do with double bit flips? you lose your block, that's what).

As far as I can tell at least the following integrity checks happen between your data and the data persistence:

• CPU L1/L2/L3 has integrity and error correction (in more and more cases -- it used to be 'special' a few decades ago)
• PCIe interconnects (and SerDes PIPE PHY blocks and other cool acronyms that boil down to 'PCIe interface') do it with CRC and retransmission
• SATA 2 and up tend to have optional error checks built in (which results in retransmission instead of correction, like PCIe)
• On-device checks where it tries to decode the payload (i.e. blocks, filesystem, sometimes even file contents), mostly on SSDs since it is used for compression, garbage collection and other housekeeping, also applies to on-device caches which themselves might be ECC-capable
• The actual storage media, specially TLC and up for NAND, and both CMR and SMR for HDDs, but also ECC (sometimes on the index flash) on modern tapes

And all of that is invisible (usually) to the OS, to the filesystem and to the applications. Mainly because everything is constantly kept secret because it's easy and cheap to just NDA the snot out of everything instead of sharing relevant parts where allowed. This also means that you cannot actually depend on it, and most drives tend to lie to the OS anyway. (if you ask a drive: are my blocks flushed to disk? it lies 9 out of 10 times AFAICT)

It used to be that the layered approach had the same benefits as the OSI model, or the smaller version of that model that we see today in ethernet/TCP/IP/HTTP etc. It meant you can swap out components without suddenly losing capabilities. The same applies to SED or FDE; you often cannot trust the implementation (especially OPAL TCG) due to the way it is integrated (usually via some dead-end IBV chain combined with random storage vendors).

This is where ZFS shines; it doesn't care how fancy the drive pretends to be, it assumes it is crap, and assumes that the more the interconnect controller tries to do, the less sure it can be about the data.

 

[gea]

There are also some legitimate reasons not to trust ZFS. The fragmentation issue ZFS suffers coupled with the variable sized blocks that require reference to block pointer trees that are often fragmented themselves will cause hdd heads to thrash during resilver if the pool has been used for any reasonable amount of time and greatly increases the chance of additional drive failures compared to "traditional" raid.

???
Any serious reference for that opinion?

Copy on Write filesystems suffer more fragmentation than older filesystems but the advanced ZFS rambased read/write cache reduces small read i/o to a fraction. It is not unusual to see 80% cache hit rate with enough RAM. The variable blocksize is a nice feature when you write datablocks < recsize. If the data to be written >=recsize, all pieces are written in recsize (up to 1M)

 

[oneplane]

???
Any serious reference for that opinion?

Copy on Write filesystems suffer more fragmentation than older filesystems but the advanced ZFS rambased read/write cache reduces small read i/o to a fraction. It is not unusual to see 80% cache hit rate with enough RAM. The variable blocksize is a nice feature when you write datablocks < recsize. If the data to be written >=recsize, all pieces are written in recsize (up to 1M)

I also don't think this is even remotely a 'trust' issue; mostly a performance consideration for which we have plenty of good mitigations and controls. It does however illustrate that if you are serious about your storage, you're also going to have to read the docs, do the work etc. so you're in a position where you can manage it in a way that matches the requirements.