520 byte drives, weird behavior formatted as 512

[Mithril]

So I'm putting together a new Truenas build, drives all from ebay. A mix of models but all HGST helium. Some care or were reformatted as 512, 2 are 520. They have all passed a full badblocks test. Some I have had for a while and were cycled out of another build, some are "new to me". Onto the weirdness.

In Truenas the 5 512 formatted drives are significantly slower (and this was not true when they were in my "bench" machine that runs Ubuntu) .
For writes:

[root@truenas ~]# for i in {0..6}; do echo $i && dd if=/dev/zero of=/dev/da$i bs=1M count=1024; done
0
1073741824 bytes transferred in 13.506698 secs (79496991 bytes/sec)
1
1073741824 bytes transferred in 13.757889 secs (78045534 bytes/sec)
2
1073741824 bytes transferred in 13.958135 secs (76925878 bytes/sec)
3
1073741824 bytes transferred in 5.416859 secs (198222223 bytes/sec)
4
1073741824 bytes transferred in 13.717107 secs (78277573 bytes/sec)
5
1073741824 bytes transferred in 5.303768 secs (202448882 bytes/sec)
6
1073741824 bytes transferred in 13.781617 secs (77911162 bytes/sec)

Reads are much more sane, roughly 202MiB-221MiB with the same 1M size.
Smaller sizes are predictably slower, but the "all reads are ok, writes are vastly asymmetric" remains.
I'm going to *guess* that I screwed up the low level formatting, but since that can take a long time and I'm not sure if I can "brick" a drive if I screw around with the command... I'm hesitant to just do so without more understanding.

All drives report " Mode: r0w0e0" from "geom disk list"
All drives report they are a mix of physical/logical 512/4096
Only 1 of the two 520 formatted drives reports "Formatted with type 1 protection | 8 bytes of protection information per logical block", but both are the identical reported size.

And I spin up a quick test Raidz1 as well as a ZFS stripe, write speeds with fully random data seem.... *fine* so.... is this just weirdness, or an actual issue?

 

[nabsltd]

And I spin up a quick test Raidz1 as well as a ZFS stripe, write speeds with fully random data seem.... *fine* so.... is this just weirdness, or an actual issue?

I had some HGST He drives that had EMC firmware (CLAR in the disk name) on them. badblocks ran about 5% slower on TrueNAS SCALE, but about 30% slower on TruNAS Core (both latest versions). But, dd and fio are about 10% faster on the EMC drives on TrueNAS SCALE. I don't know what usage pattern is assumed in the firmware, but the speeds are fine in real-world use, so I'm keeping them instead of trying a return.

So, check the firmware and see if you can update, since that should at least make the drives perform the same. If you can't because of locked firmware, then you'll probably have to live with what you have.

 

[Mithril]

Hmm, and I remembering the `format to 512 bytes` command wrong? did a format on the two drives reporting a smaller side and they did not change size