Seagate X12 HDDs hanging on by a thread

[Railgun]

Multiply by 22 and these disks most definitely have an issue.

Error counter log:
Errors Corrected Total Correction Gigabytes Total
by ECC rereads/ errors algorithm processed uncorrected
fast| delayed rewrites corrected invocations [10^9 bytes] errors
read: 2136549412 0 0 2136549412 0 26358.286 0
write: 0 0 0 0 0 517.500 0

Non-medium error count: 0

I'd posted elsewhere about rejigging my NAS with potentially known suspect disks.

Write performance

root@truenas[/mnt/TheEndHouse]# dd if=/dev/zero of=/mnt/TheEndHouse/ddfile bs=2048k count=100000
100000+0 records in
100000+0 records out
209715200000 bytes transferred in 91.586038 secs (2289816280 bytes/sec)

Read performance with the cache disabled (cancelled as it was taking forever)

root@truenas[/mnt/TheEndHouse]# dd of=/dev/null if=/mnt/TheEndHouse/ddfile bs=2048k count=100000
30813+0 records in
30813+0 records out
64619544576 bytes transferred in 771.877023 secs (83717409 bytes/sec)

While it's been long known that this series was problematic, I'd never seen anything that indicated why. This isn't necessarily the smoking gun, but it's certainly not benefiting the thing at all.

 

[mr44er]

1. dd is really bad to check disk performance. It can be used as a first check, but for any serious benching it's just the wrong tool.
2. You do R/W on the pool, you should do this on every single disk.
3. 2289816280 bytes/sec = 2,2GB/s can't be right. Only really big pools with many vdevs can do this. lz4 or zstd here just compresses the zeroes ultrafast and that is regardless of any cache you deactivate/active on the disks.
4. A high error (and correction) rate is normal for Seagate disks. This is not a fault, that's just their way being more verbose. I forgot the details, but you can google it.
5. If your truenas has still FreeBSD under the hood, do diskinfo -citv /dev/da$ for every disk and compare. A spinning disk has a plausible, realistic sequential range from 100 - ~180 MB/s and that's it. Really new models or fast (15k) can reach near the 250MB/s under some circumstances but that's really the possible top max.

 

[Railgun]

1. Fair enough. Any other testing will given similar results...

2. as I'm testing pool performance as a whole.

3. 11x mirrored vdevs. Big enough? Compression was disabled for the testing as was the cache. 110-250MBps from inside to outside, so 11x vdevs...

4. Considering it increments on reads, I would never expect ANY corrected anything unless there was a legit issue. I appreciate Seagate have indicated they report "raw error values," that does not sit well with me given the performance as well as the history for this line, I would suggest this is a problem.

5. It is, it's Core.

That said, regardless of the testing method, I'd not expect reads on the pool to be as bad as they are, provided there is/are no actual disk problem/s.

 

[mr44er]

1. / 2. Pls repeat the test with anything more near the physical sector size of a single disk and the blocksize of your pool and /dev/random. That said, 1x with /dev/random bs=4096 (no 'k'), 1x with bs=128k and 1x with bs=1M

3. Ok, expecting sequential write of 150MB/s for single disk would be 11*150 -> ~1,65GB/s to the pool and seq. read @200 would be 22*200 -> ~4,4GB/s

4. I and others can tell, that's just how they behave and do things. I saw this behaviour on all Seagate models I ever owned.

5. If diskinfo shows I/O and throughput range for every disk within specs, your performance problem is buried elsewhere.

Open on a console gstat, repeat your tests and keep an eye on the disks. Any sequential run should push them to ~100% load and red color. A diskinfo run pushes them to the max, if your dd-tests not, then your problem is elsewhere, but not the disks itself.

 

[Railgun]

Well, as we're going down this road...

I'm going to skip the random test as it's not relevant in this instance. gstat was for sequental tests on a 13GB file. Needless to say reads are a snapshot and vary. Diskinfo highlights some outlyers, but the only thing that really stands out is da18's transfer rate. Da9 and 21 are spares.

For brevity, here's the output from some testing

NAS Disks.xlsx

Diskinfo Disk Performance,da1,da2,da3,da4,da5,da6,da7,da8,da9,da10,da11,da12,da13,da14,da15,da16,da17,da18,da19 I/O Overhead (msec/sector) time to read 10MB block,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.003,0.002,0.003,0.003 time to read 20480 ...

docs.google.com

On top of that...

Adapter Vendor Device SubSys SubSys
Index Type ID ID Pci Address Ven ID Dev ID
----- ------------ ------ ------ ----------------- ------ ------
0 SAS3224 1000h c4h 00h:04h:00h:00h 1000h 31a0h

Adapter Vendor Device SubSys SubSys
Index Type ID ID Pci Address Ven ID Dev ID
----- ------------ ------ ------ ----------------- ------ ------
1 SAS3224 1000h c4h 00h:1bh:00h:00h 1000h 31a0h

As depicted in the spreadsheet, the mirrors are split across HBAs. This was done as the initial setup that had these disks on a single controller would give me a massive amount of read/write/verify errors across multiple disks constantly. I'd pulled them all out and during some additional testing against one of the HBAs, I was getting similar errors until I reduced the number of disks.

Truenas in a VM under ESXi 8. HBAs passed through with the interface via SR-IOV (25Gb Intel E810).

The ONLY thing that's different with any performance testing is the additional HBA.

Underlying HW is an Epyc 7543 - 16 CPUs and 256G to the VM. HBAs are on PCI-E lanes to the same CCD.

I have a 3x NVMe mirror for a special/metadata vdev, and Truenas has asked to open a ticket as it's not quite behaving as I'd expect.

I should say that in the previous setup, a 3 disk x 7vdev z1 on a single controller performed significantly better. This was change was done for capacity and the aforementioned oddity with the disks against the single controller. Performance has slowly degraded but with the initial new setup, performance wasn't great. The X12s also have some apparent known slow down issue as they get full, but we're just around 55% utilization.

 

[mr44er]

Ok, I have no idea how to proceed further.

To 100% rule out the config and/or (not) nail down to the disks, testing with 22 other disks would be good, but I guess that's impossible.