Proxmox - ZFS Disk Failure vs Smartctl Report

gb00s

Active Member
Jul 25, 2018
223
61
28
Malta
I'm experiencing a ZFS disk failure (1) for the first time since I'm working with ZFS. This disk is in a Raid10 pool. The strange thing is that different smartctl reports were given during the last couple of weeks. Sometimes with an error. Sometimes without an error. I always just made short background tests. Maybe a mistake. Then I could easily scrub, clear out ZFS errors while the disk wasn't shown as 'faulty' in ZFS and all was fine. But this time ZFS shows the disk is in a 'faulty' state and the vpool 'degraded'. I just started a final long background check.

ZFS_DiskFailure1(scaled).png

Another smartctl report by Proxmox show no errors at all.

ZFS_DiskFailure2(scaled).png

If the final smartctl report is negative again and without errors, do you still prefer to replace the disk, or I may have to investigate the HBA as well, which might throw some error? Or what else would you check and try in order to avoid 'false positives' and a disk exchange? Any opinion and/or comment is appreciated.

Thank you in advance.

Mike
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
4,592
912
113
You could run badblocks on it (non destructive if you dont have a spare/no option to evacuate the data).

If its always the same then maybe swap drives to see if it moves with it?
 
  • Like
Reactions: gb00s

ttabbal

Active Member
Mar 10, 2016
757
202
43
43
For me, it depends on the type of error smart reports. I had one drive report transport CRC errors, swapping it to another slot fixed the problem. I suspect the cable or backplane has an issue on that port, but I haven't taken the time to investigate further. A media error that happens more than once, I swap the drive with a new one and do badblocks or similar on it to allow the controller to reallocate blocks. Sometimes that clears it, but I've always had it be a short term fix. Usually, a write error on the media that you see means the controller is out of spares. On the upside, 3TB drives are pretty cheap now, so at least it's not expensive to have a spare. I find that with media tests, I rarely get a false positive if I consider the report a sign of impending failure. I can usually get a little more life out of them, but they are not long for this world. So once I start getting them I assume I need a tested, ready to replace drive on hand. Testing a disk usually takes a few days for me, so I like to order early. I don't have a bunch of spares collecting dust, so it works for me.

For testing, I like to do short tests with long tests mixed in on a cycle. That helps catch errors before they become a big deal. Along with ZFS scrubs to validate the data.

I might be paranoid, but I assume drives not only will fail eventually, but are evil and actively out to get me. :)
 
  • Like
Reactions: gb00s

gb00s

Active Member
Jul 25, 2018
223
61
28
Malta
Ok, thanks for both of you guys @Rand__ @ttabbal

I have 3-4 spare drives. Ok I will make a scrub, clear the errors and will move the drives then. After that I will make a badblocks test with '-nsv' option and check HBA + cable. Then probably replace the drive if I see errors again. Just dry testing how to replace it within ZFS as this is my first one. Thanks again and have a nice evening.
 

gb00s

Active Member
Jul 25, 2018
223
61
28
Malta
In a first badblocks -v (read-only) test there are no errors. Will run a badblocks -nsv ... from tomorrow over the weekend. Smartctl long-term also gave no errors again. For now just cleared the errors and made a scrub. Everything is online and normal again.
 

gb00s

Active Member
Jul 25, 2018
223
61
28
Malta
I'm losing faith and I'm no longer sure what to do. Tonight on another machine at home I got the following:

[177447.790023] scsi_io_completion_action: 121 callbacks suppressed
[177447.790027] sd 0:0:0:0: [sda] tag#8389 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177447.790029] sd 0:0:0:0: [sda] tag#8389 Sense Key : Aborted Command [current]
[177447.790032] sd 0:0:0:0: [sda] tag#8389 Add. Sense: Nak received
[177447.790034] sd 0:0:0:0: [sda] tag#8389 CDB: Read(10) 28 00 14 04 aa 28 00 00 a0 00
[177447.790034] print_req_error: 121 callbacks suppressed
[177447.790036] blk_update_request: I/O error, dev sda, sector 335850024 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177447.790338] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171954163712 size=81920 flags=1808b0
[177447.794672] sd 0:0:0:0: [sda] tag#8401 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177447.794683] sd 0:0:0:0: [sda] tag#8401 Sense Key : Aborted Command [current]
[177447.794685] sd 0:0:0:0: [sda] tag#8401 Add. Sense: Nak received
[177447.794719] sd 0:0:0:0: [sda] tag#8401 CDB: Read(10) 28 00 14 04 ac c8 00 01 00 00
[177447.794726] blk_update_request: I/O error, dev sda, sector 335850696 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177447.795123] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171954507776 size=131072 flags=1808b0
[177447.827437] sd 0:0:0:0: [sda] tag#8409 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177447.827445] sd 0:0:0:0: [sda] tag#8409 Sense Key : Aborted Command [current]
[177447.827449] sd 0:0:0:0: [sda] tag#8409 Add. Sense: Nak received
[177447.827462] sd 0:0:0:0: [sda] tag#8409 CDB: Read(10) 28 00 14 04 ae 20 00 01 58 00
[177447.827465] blk_update_request: I/O error, dev sda, sector 335851040 op 0x0:(READ) flags 0x700 phys_seg 5 prio class 0
[177447.827824] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171954683904 size=176128 flags=40080cb0
[177447.953148] sd 0:0:6:0: [sdg] tag#8393 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177447.953158] sd 0:0:6:0: [sdg] tag#8393 Sense Key : Aborted Command [current]
[177447.953173] sd 0:0:6:0: [sdg] tag#8393 Add. Sense: Nak received
[177447.953177] sd 0:0:6:0: [sdg] tag#8393 CDB: Read(10) 28 00 15 03 31 78 00 01 00 00
[177447.953180] blk_update_request: I/O error, dev sdg, sector 352530808 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177447.953477] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180494725120 size=131072 flags=1808b0
[177447.956638] sd 0:0:6:0: [sdg] tag#8402 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177447.956641] sd 0:0:6:0: [sdg] tag#8402 Sense Key : Aborted Command [current]
[177447.956643] sd 0:0:6:0: [sdg] tag#8402 Add. Sense: Nak received
[177447.956644] sd 0:0:6:0: [sdg] tag#8402 CDB: Read(10) 28 00 15 03 32 78 00 01 00 00
[177447.956646] blk_update_request: I/O error, dev sdg, sector 352531064 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177447.957186] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180494856192 size=131072 flags=1808b0
[177448.065861] sd 0:0:1:0: [sdb] tag#8387 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177448.065868] sd 0:0:1:0: [sdb] tag#8387 Sense Key : Aborted Command [current]
[177448.065882] sd 0:0:1:0: [sdb] tag#8387 Add. Sense: Nak received
[177448.065884] sd 0:0:1:0: [sdb] tag#8387 CDB: Read(10) 28 00 13 00 d0 28 00 07 c8 00
[177448.065886] blk_update_request: I/O error, dev sdb, sector 318820392 op 0x0:(READ) flags 0x700 phys_seg 17 prio class 0
[177448.066221] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163234992128 size=1019904 flags=40080cb0
[177448.072221] sd 0:0:1:0: [sdb] tag#8388 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177448.072223] sd 0:0:1:0: [sdb] tag#8388 Sense Key : Aborted Command [current]
[177448.072225] sd 0:0:1:0: [sdb] tag#8388 Add. Sense: Nak received
[177448.072227] sd 0:0:1:0: [sdb] tag#8388 CDB: Read(10) 28 00 13 00 d7 f0 00 07 70 00
[177448.072228] blk_update_request: I/O error, dev sdb, sector 318822384 op 0x0:(READ) flags 0x700 phys_seg 17 prio class 0
[177448.072677] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163236012032 size=974848 flags=40080cb0
[177448.121551] sd 0:0:7:0: [sdh] tag#8420 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177448.121560] sd 0:0:7:0: [sdh] tag#8420 Sense Key : Aborted Command [current]
[177448.121564] sd 0:0:7:0: [sdh] tag#8420 Add. Sense: Nak received
[177448.121566] sd 0:0:7:0: [sdh] tag#8420 CDB: Read(10) 28 00 15 02 f8 d0 00 08 00 00
[177448.121569] blk_update_request: I/O error, dev sdh, sector 352516304 op 0x0:(READ) flags 0x700 phys_seg 16 prio class 0
[177448.121895] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180487299072 size=1048576 flags=40080cb0
[177448.123010] sd 0:0:0:0: [sda] tag#8397 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177448.123012] sd 0:0:0:0: [sda] tag#8397 Sense Key : Aborted Command [current]
[177448.123013] sd 0:0:0:0: [sda] tag#8397 Add. Sense: Nak received
[177448.123014] sd 0:0:0:0: [sda] tag#8397 CDB: Read(10) 28 00 14 04 b2 c0 00 01 00 00
[177448.123016] blk_update_request: I/O error, dev sda, sector 335852224 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177448.123385] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171955290112 size=131072 flags=1808b0
[177448.131742] sd 0:0:6:0: [sdg] tag#8425 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177448.131748] sd 0:0:6:0: [sdg] tag#8425 Sense Key : Aborted Command [current]
[177448.131769] sd 0:0:6:0: [sdg] tag#8425 Add. Sense: Nak received
[177448.131787] sd 0:0:6:0: [sdg] tag#8425 CDB: Read(10) 28 00 15 03 3b 18 00 01 00 00
[177448.131810] blk_update_request: I/O error, dev sdg, sector 352533272 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177448.136191] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180495986688 size=131072 flags=1808b0
[177448.229141] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180496379904 size=131072 flags=1808b0
[177448.254215] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180488613888 size=942080 flags=40080cb0
[177448.259107] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180489555968 size=245760 flags=40080cb0
[177448.273457] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180496642048 size=573440 flags=40080cb0
[177448.364012] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180489904128 size=978944 flags=40080cb0
[177448.435751] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171955683328 size=131072 flags=1808b0
[177448.461833] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171956002816 size=102400 flags=1808b0
[177448.484617] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180498157568 size=131072 flags=1808b0
[177448.489828] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94564126720 size=1048576 flags=40080cb0
[177448.493162] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180491292672 size=1007616 flags=40080cb0
[177448.497234] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94565175296 size=1048576 flags=40080cb0
[177448.499620] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180498735104 size=135168 flags=40080cb0
[177448.505284] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180498870272 size=131072 flags=1808b0
[177448.542736] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171956117504 size=131072 flags=1808b0
[177448.549954] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171956248576 size=503808 flags=40080cb0
[177448.590867] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94566223872 size=397312 flags=40080cb0
[177448.598276] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94566674432 size=937984 flags=40080cb0
[177448.641854] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180492627968 size=626688 flags=40080cb0
[177448.644062] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180499263488 size=131072 flags=1808b0
[177448.696164] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94567612416 size=540672 flags=40080cb0
[177448.759248] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171957833728 size=131072 flags=1808b0
[177448.781792] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180493373440 size=1044480 flags=40080cb0
[177448.788051] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180494462976 size=655360 flags=40080cb0
[177448.801030] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94568280064 size=753664 flags=40080cb0
[177448.806767] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180500811776 size=397312 flags=40080cb0
[177448.808070] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94569103360 size=794624 flags=40080cb0
[177448.925013] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180495204352 size=1044480 flags=40080cb0
[177448.932108] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180496248832 size=966656 flags=40080cb0
[177448.941399] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94569897984 size=131072 flags=1808b0
[177448.944793] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180501209088 size=131072 flags=1808b0
[177448.948304] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94570037248 size=667648 flags=40080cb0
[177449.044075] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180501602304 size=131072 flags=1808b0
[177449.045074] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180497502208 size=1048576 flags=40080cb0
[177449.058938] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171949862912 size=978944 flags=40080cb0
[177449.062038] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180501745664 size=1036288 flags=40080cb0
[177449.080300] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171958251520 size=188416 flags=40080cb0
[177449.091200] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94570852352 size=995328 flags=40080cb0
[177449.095358] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180502781952 size=98304 flags=1808b0
[177449.095403] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94571954176 size=921600 flags=40080cb0
[177449.127444] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163238035456 size=1036288 flags=40080cb0
[177449.153370] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180498550784 size=974848 flags=40080cb0
[177449.153372] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180502966272 size=94208 flags=1808b0
[177449.156429] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171950841856 size=925696 flags=40080cb0
[177449.158219] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94564126720 size=1048576 flags=40080cb0
[177449.160328] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180503060480 size=798720 flags=40080cb0
[177449.169164] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180503859200 size=524288 flags=40080cb0
[177449.176200] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171958439936 size=524288 flags=40080cb0
[177449.241910] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163239071744 size=987136 flags=40080cb0
[177449.254481] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94572875776 size=212992 flags=40080cb0
[177449.258605] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171951767552 size=1048576 flags=40080cb0
[177449.262030] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180499525632 size=1024000 flags=40080cb0
[177449.264376] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94565175296 size=1048576 flags=40080cb0
[177449.268665] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180500549632 size=921600 flags=40080cb0
[177449.275566] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180504514560 size=131072 flags=1808b0
[177449.282224] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180504645632 size=978944 flags=40080cb0
[177449.287513] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180505624576 size=262144 flags=40080cb0
[177449.365239] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171959488512 size=393216 flags=40080cb0
[177449.384141] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94566223872 size=397312 flags=40080cb0
[177449.419692] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180501209088 size=131072 flags=1888b1
[177449.456743] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171953856512 size=1003520 flags=40080cb0
[177449.463582] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171954860032 size=1007616 flags=40080cb0
[177449.496149] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180501471232 size=1015808 flags=40080cb0
[177449.506854] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94566674432 size=937984 flags=40080cb0
[177449.513346] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94567612416 size=540672 flags=40080cb0
[177449.567526] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171959881728 size=131072 flags=1808b0
[177449.578628] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171960143872 size=1048576 flags=40080cb0
[177449.685371] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180506578944 size=794624 flags=40080cb0
[177449.693337] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180502487040 size=978944 flags=40080cb0
[177449.695580] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94573219840 size=876544 flags=40080cb0
[177449.701334] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180503465984 size=1048576 flags=40080cb0
[177449.770951] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171962241024 size=131072 flags=1808b0
[177449.807728] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180504514560 size=978944 flags=40080cb0
[177449.813773] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180505493504 size=393216 flags=40080cb0
[177449.825041] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180508160000 size=352256 flags=40080cb0
[177449.859801] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171955867648 size=995328 flags=40080cb0
[177449.950781] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163240058880 size=1015808 flags=40080cb0
[177449.957332] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163241074688 size=1044480 flags=40080cb0
[177449.960209] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171956862976 size=970752 flags=40080cb0
[177449.962682] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171962896384 size=131072 flags=1808b0
[177449.964365] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171963158528 size=131072 flags=1808b0
[177449.984759] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171963289600 size=131072 flags=1808b0
[177449.991268] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171963420672 size=393216 flags=40080cb0
[177450.061991] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163242119168 size=995328 flags=40080cb0
[177450.303635] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171957833728 size=999424 flags=40080cb0
[177450.310172] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171963944960 size=212992 flags=40080cb0
[177450.344011] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180509630464 size=131072 flags=1808b0
[177450.401942] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171958833152 size=1048576 flags=40080cb0
[177450.507415] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171965829120 size=110592 flags=1808b0
[177450.516962] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171959881728 size=1048576 flags=40080cb0
[177450.531575] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171966128128 size=131072 flags=1808b0
[177450.632516] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171961978880 size=1048576 flags=40080cb0
[177450.664542] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163243130880 size=1048576 flags=40080cb0
[177450.707010] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94574239744 size=987136 flags=40080cb0
[177450.739201] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171963027456 size=1048576 flags=40080cb0
[177450.745228] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171964076032 size=700416 flags=40080cb0
[177450.773691] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163244232704 size=868352 flags=40080cb0
[177450.809389] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94575337472 size=921600 flags=40080cb0
[177450.815494] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94576259072 size=393216 flags=40080cb0
[177450.861564] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7371f-part1 error=5 type=1 offset=171965698048 size=561152 flags=40080cb0
[177451.208906] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94569103360 size=925696 flags=40080cb0
[177451.321037] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94570037248 size=667648 flags=40080cb0
[177451.434807] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163245752320 size=491520 flags=40080cb0
[177451.443286] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163246288896 size=503808 flags=40080cb0
[177451.458910] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94569369600 size=131072 flags=1888b1
[177451.544709] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=163246850048 size=815104 flags=40080cb0
[177451.582481] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94570852352 size=995328 flags=40080cb0
[177451.590831] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94571954176 size=921600 flags=40080cb0
[177451.694874] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94572875776 size=1007616 flags=40080cb0
[177451.701023] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94573883392 size=1011712 flags=40080cb0
[177452.000040] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180506578944 size=925696 flags=40080cb0
[177452.053876] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94574895104 size=331776 flags=40080cb0
[177452.106438] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180508512256 size=974848 flags=40080cb0
[177452.112646] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180509487104 size=667648 flags=40080cb0
[177452.175841] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94575337472 size=921600 flags=40080cb0
[177452.935200] scsi_io_completion_action: 114 callbacks suppressed
[177452.935211] sd 0:0:2:0: [sdc] tag#8415 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177452.935225] sd 0:0:2:0: [sdc] tag#8415 Sense Key : Aborted Command [current]
[177452.935228] sd 0:0:2:0: [sdc] tag#8415 Add. Sense: Nak received
[177452.935231] sd 0:0:2:0: [sdc] tag#8415 CDB: Read(10) 28 00 0b 02 9f 98 00 03 00 00
[177452.935243] print_req_error: 114 callbacks suppressed
[177452.935246] blk_update_request: I/O error, dev sdc, sector 184721304 op 0x0:(READ) flags 0x700 phys_seg 6 prio class 0
[177452.935554] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6b013-part1 error=5 type=1 offset=94576259072 size=393216 flags=40080cb0
[177453.201582] sd 0:0:1:0: [sdb] tag#8394 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.201591] sd 0:0:1:0: [sdb] tag#8394 Sense Key : Aborted Command [current]
[177453.201605] sd 0:0:1:0: [sdb] tag#8394 Add. Sense: Nak received
[177453.201607] sd 0:0:1:0: [sdb] tag#8394 CDB: Read(10) 28 00 8b b9 c4 20 00 00 e0 00
[177453.201610] blk_update_request: I/O error, dev sdb, sector 2344207392 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177453.201943] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6ff53-part1 error=5 type=1 offset=1200233136128 size=114688 flags=80bc0
[177453.221082] sd 0:0:4:0: [sde] tag#8421 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.221084] sd 0:0:4:0: [sde] tag#8421 Sense Key : Aborted Command [current]
[177453.221085] sd 0:0:4:0: [sde] tag#8421 Add. Sense: Nak received
[177453.221087] sd 0:0:4:0: [sde] tag#8421 CDB: Read(10) 28 00 8b b9 c6 20 00 00 e0 00
[177453.221088] blk_update_request: I/O error, dev sde, sector 2344207904 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177453.221386] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=1200233398272 size=114688 flags=80bc0
[177453.276266] sd 0:0:4:0: [sde] tag#8412 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.276271] sd 0:0:4:0: [sde] tag#8412 Sense Key : Aborted Command [current]
[177453.276273] sd 0:0:4:0: [sde] tag#8412 Add. Sense: Nak received
[177453.276275] sd 0:0:4:0: [sde] tag#8412 CDB: Read(10) 28 00 0b 02 a2 98 00 01 00 00
[177453.276278] blk_update_request: I/O error, dev sde, sector 184722072 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177453.276631] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071772837-part1 error=5 type=1 offset=94576652288 size=131072 flags=1808b0
[177453.281740] sd 0:0:7:0: [sdh] tag#8414 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.281744] sd 0:0:7:0: [sdh] tag#8414 Sense Key : Aborted Command [current]
[177453.281751] sd 0:0:7:0: [sdh] tag#8414 Add. Sense: Nak received
[177453.281755] sd 0:0:7:0: [sdh] tag#8414 CDB: Read(10) 28 00 15 03 a7 30 00 01 00 00
[177453.281759] blk_update_request: I/O error, dev sdh, sector 352560944 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177453.282433] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6d1ab-part1 error=5 type=1 offset=180510154752 size=131072 flags=1808b0
[177453.284244] sd 0:0:6:0: [sdg] tag#8424 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.284245] sd 0:0:6:0: [sdg] tag#8424 Sense Key : Aborted Command [current]
[177453.284247] sd 0:0:6:0: [sdg] tag#8424 Add. Sense: Nak received
[177453.284249] sd 0:0:6:0: [sdg] tag#8424 CDB: Read(10) 28 00 15 03 ae 30 00 00 e0 00
[177453.284251] blk_update_request: I/O error, dev sdg, sector 352562736 op 0x0:(READ) flags 0x700 phys_seg 3 prio class 0
[177453.284658] sd 0:0:0:0: [sda] tag#8423 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.284660] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c50071b7425f-part1 error=5 type=1 offset=180511072256 size=114688 flags=1808b0
[177453.284661] sd 0:0:0:0: [sda] tag#8423 Sense Key : Aborted Command [current]
[177453.284663] sd 0:0:0:0: [sda] tag#8423 Add. Sense: Nak received
[177453.284665] sd 0:0:0:0: [sda] tag#8423 CDB: Read(10) 28 00 14 05 0d e0 00 01 00 00
[177453.284667] blk_update_request: I/O error, dev sda, sector 335875552 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[177453.285074] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171967234048 size=131072 flags=1808b0
[177453.287448] sd 0:0:0:0: [sda] tag#8425 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.287450] sd 0:0:0:0: [sda] tag#8425 Sense Key : Aborted Command [current]
[177453.287452] sd 0:0:0:0: [sda] tag#8425 Add. Sense: Nak received
[177453.287453] sd 0:0:0:0: [sda] tag#8425 CDB: Read(10) 28 00 14 05 08 70 00 05 70 00
[177453.287455] blk_update_request: I/O error, dev sda, sector 335874160 op 0x0:(READ) flags 0x700 phys_seg 13 prio class 0
[177453.287887] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171966521344 size=712704 flags=40080cb0
[177453.290651] sd 0:0:0:0: [sda] tag#8391 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.290653] sd 0:0:0:0: [sda] tag#8391 Sense Key : Aborted Command [current]
[177453.290654] sd 0:0:0:0: [sda] tag#8391 Add. Sense: Nak received
[177453.290656] sd 0:0:0:0: [sda] tag#8391 CDB: Read(10) 28 00 14 05 0e e0 00 08 00 00
[177453.290657] blk_update_request: I/O error, dev sda, sector 335875808 op 0x0:(READ) flags 0x700 phys_seg 16 prio class 0
[177453.291040] zio pool=poolSG vdev=/dev/disk/by-id/scsi-35000c5006bb6f9c3-part1 error=5 type=1 offset=171967365120 size=1048576 flags=40080cb0
[177453.291441] sd 0:0:6:0: [sdg] tag#8384 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[177453.291450] sd 0:0:6:0: [sdg] tag#8384 Sense Key : Aborted Command [current]
[177453.291484] sd 0:0:6:0: [sdg] tag#8384 Add. Sense: Nak received
[177453.291508] sd 0:0:6:0: [sdg] tag#8384 CDB: Read(10) 28 00 15 03 af 10 00 06 e0 00
[177453.291532] blk_update_request: I/O error, dev sdg, sector 352562960 op 0x0:(READ) flags 0x700 phys_seg 15 prio class 0
poolSG_Errors1(scaled).png

From a zpool status -v I could see all these drives were marked as 'FAILED'. I cleared the errors, got resilvered, scrubbed it and now I still get these errors. These are all Seagate 1.2TB's I already wrote about some weeks ago which were going 'hot' most of the time. Whenever I have errors on these disk the temps on the disks are going > 50C. The drives were just in idle. No write and or heavy read operations when the errors occurred.

root@pve2:~# dmesg | less
root@pve2:~# zpool status -v
pool: poolSG
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: ZFS Message ID: ZFS-8000-8A
scan: scrub repaired 533M in 0 days 00:03:51 with 14 errors on Sat Sep 26 11:08:53 2020
config:

NAME STATE READ WRITE CKSUM
poolSG DEGRADED 0 0 0
mirror-0 DEGRADED 8.47K 0 0
scsi-35000c50071b145b7 FAULTED 877 0 133 too many errors
scsi-35000c5006bb6ff53 ONLINE 9.95K 0 1.11K
mirror-1 ONLINE 4.59K 0 0
scsi-35000c5006bb6b013 ONLINE 5.69K 11 788
scsi-35000c50071772837 ONLINE 5.90K 1 845
mirror-2 ONLINE 3.62K 0 0
scsi-35000c5006bb6f9c3 ONLINE 4.75K 0 724
scsi-35000c50071b7371f ONLINE 5.89K 15 821
mirror-3 ONLINE 3.81K 0 0
scsi-35000c50071b7425f ONLINE 5.07K 0 671
scsi-35000c5006bb6d1ab ONLINE 5.60K 4 781

errors: Permanent errors have been detected in the following files:

/poolSG/gnops/live/gnops/Finance/....msg
/poolSG/gnops/live/gnops/Marketing Material/....jpg
/poolSG/gnops/live/gnops/Marketing Material/....jpg
/poolSG/gnops/live/gnops/Marketing Material/....jpg
/poolSG/gnops/live/gnops/Finance/.....pdf
/poolSG/gnops/live/gnops/Clients/....png
/poolSG/gnops/live/gnops/Clients/....png
/poolSG/gnops/live/gnops/Suppliers/....pdf
/poolSG/gnops/live/gnops/Clients/....pdf
/poolSG/gnops/live/gnops/Marketing Material/....jpg
/poolSG/gnops/live/gnops/Marketing Material/....jpg
/poolSG/gnops/live/gnops/Agencies/....jpg
/poolSG/gnops/live/gnops/Clients/....pdf
/poolSG/gnops/live/gnops/Marketing Material/....mp4

pool: poolVM
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: ZFS Message ID: ZFS-8000-9P
scan: scrub repaired 0B in 0 days 00:00:01 with 0 errors on Sat Sep 26 11:13:40 2020
config:

NAME STATE READ WRITE CKSUM
poolVM ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000039648085878 ONLINE 1 0 0
scsi-35000039638115ad8 ONLINE 1 0 0
mirror-1 ONLINE 0 0 0
scsi-35000039638115a50 ONLINE 0 0 0
scsi-35000039648085e20 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
scsi-35000039668139f40 ONLINE 0 0 0
scsi-350000395c8398d44 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
scsi-350000396681397c4 ONLINE 0 0 0
scsi-3500003966813a238 ONLINE 0 0 0

errors: No known data errors
root@pve2:~#
poolSG_Errors(scaled).png

Even the poolVM I just build without data already shows an error alert I can't get rid of. These are totally different disks.

While it seems to build other problems on the other and newly build poolVM, do I have a disk problem here or is it an HBA or backplate problem? HBA is a flashed DELL Perc H310. The backplane is a BPN-SAS2-216EL2. I'm asking as the errors are all coming at once and I never experienced the same issue with failing disks all at the same time. I'm working to get these disk running without problems for weeks. No luck so far. Bought everything used from the same guy here. Always gives me a headache. Software/OS wise I don't think it's a Proxmox issue.

What else can I do? Changing HBA ... yes. Changing backplane ... no. Changing disks ... no. Or can it be something else? These files are synced via 'Syncthing' from a WS2019 SMB-share in realtime. Shall I take the backups, delete the pool, test the drives with badblocks, build a new pool? Can this also be a settings issue?

Sorry if I sound a little dramatic, but I can't get these f.... parts working and just go to sleep. I need this running flawlessly, but always have problems. I'm close to throw the whole server with all the parts away and take the loss.

EDIT1: So just rebooted and the errors I/O-errors are still there.

poolSG_Errors2(scaled).png
 
Last edited:

ttabbal

Active Member
Mar 10, 2016
757
202
43
43
I've never seen that sort of issue. My thought would be to get a breakout cable so you can connect the drives directly to the HBA. Then do a test, perhaps with just one drive, running it hard. The idea is get a baseline that works so you know where to look. It could even be an older or unused drive. If you have one available, I think I might start with a known good computer as well, just in case you have some hardware issue on that server. Once you get a working test bed, try one of the drives giving you issues. If it works there, you can try putting it in the server. Changing one thing at a time, so move the HBA, cable, and drive that works to the server and run tests on it there. I would disconnect all other drives to ensure they can't get messed up somehow from testing. If it works with cables, try the backplane.

If you only get issues in the server, don't discount a power supply issue. I had a weak PSU that caused weird errors. 99% of the time, they just don't work, but they can be REALLY annoying if you get a 1%er. :)

It might be a good idea to go back to basics and do a full burn in test on the hardware. Memtest86, Prime95, linpack, etc.. Just to run everything and try to break it. I like to run each one for 24 hours minimum. I've actually had tests run for >12 hours before failing. The drives could also be having issues from running too hot over time. Diagnosing this stuff can take a long time when it doesn't instantly fail.

With all the drives reporting issues at the same time, it makes me lean toward the HBA, cables, or backplane. But it could also be something lower level.
 

gb00s

Active Member
Jul 25, 2018
223
61
28
Malta
@ttabbal

Thank you for sharing. But I just pulled the drives out today.

I can write down what I have done so far, which brings me to the conclusion either the drives weren't working or the drives weren't fine with the backplane (firmware issues??). Why? Because other drives are working fine on the same backplane, on the same cables and on the same HBA's.

I tested over the weeks:

1. PSUs as you suggested ... I tested 5 different PSU's in the last days ... 3x Supermicros + 2Seasonics (ATX)
2. Motherboards ... 5x ... Asus Z10PA-D8 / SM X10DLR-CLN4 / SM x10SMSLL-F
3. HBAs ... 5x ... 2x Perc H310 / 2x LSI 9211-8i / 1x HP H220
4. Cables... Several SFF-8087 to (4x) SFF-8482 and SFF-8087 end-to-end
5. Changed disks to HGST HUC101212CSS600 and HGST HUC109090CSS600
6. Changed ZFS RAID configs

The only thing I could not test and switch was the backplane. But I had 8 drives switched over to a fully working server in the company over the weekend. They through errors after 5h40min. So I immediately gave up after the first errors. Different HBA, different PSU, different cables different CPUs ... so totally different environment. Logs always show the same error from above. Always with ZFS filesystem. Never tested LVM or plain partition with just simple XFS which was my main filesystem for years. The other drives from above did not through errors. I also have some Toshiba 900GB SAS SSF 2.5 (NetApp) which through errors just after a reformat of the drive. I had to reformat them again and they are fine since. I did the same with the Seagates without success.

The seller here was so kind as to fully refund me. A mega 'thanks' for that. He already asked me for weeks to refund. I always rejected until now after I see no other options why these drives always through errors.

Thanks to all.