Added new disks and pool turned out misaligned

[eroji]

I swapped out 4 of my older HGST 4TB drives for new Toshiba 5TBs in my FreeNAS server, which is serving backend storage for my 2 ESXi servers over iSCSI. After the drive swaps completed, I noticed a problem with the performance. I ran a quick CrystalDiskMark to see if there were any improvements and I saw no only it did not improve, it dropped the 4K results significantly. From 250MB/s 4kQD32 read down to 10MB/s for instance. After some Googling and asking around on FreeNAS forum and Reddit, it turns out my pool was misaligned.

I am not entirely sure if this had been there all along (likely) but now I am looking for some suggestions.

First, my current pool consists of

    da0 at mps0 bus 0 scbus0 target 8 lun 0
    da0: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da0: Serial Number         74D3K4ICFS9A
    da0: 600.000MB/s transfers
    da0: Command Queueing enabled
    da0: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da1 at mps0 bus 0 scbus0 target 9 lun 0
    da1: <ATA TOSHIBA MD04ACA5 FP1A> Fixed Direct Access SCSI-6 device
    da1: Serial Number         54O2K1XQFS9A
    da1: 600.000MB/s transfers
    da1: Command Queueing enabled
    da1: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da2 at mps0 bus 0 scbus0 target 11 lun 0
    da2: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da2: Serial Number         84P1K4MVFS9A
    da2: 600.000MB/s transfers
    da2: Command Queueing enabled
    da2: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da3 at mps0 bus 0 scbus0 target 12 lun 0
    da3: <ATA TOSHIBA MD04ACA5 FP1A> Fixed Direct Access SCSI-6 device
    da3: Serial Number         649FK2WCFS9A
    da3: 600.000MB/s transfers
    da3: Command Queueing enabled
    da3: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da4 at mps0 bus 0 scbus0 target 14 lun 0
    da4: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da4: Serial Number         74D3K4IGFS9A
    da4: 600.000MB/s transfers
    da4: Command Queueing enabled
    da4: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da5 at mps0 bus 0 scbus0 target 17 lun 0
    da5: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da5: Serial Number         84P1K4MOFS9A
    da5: 600.000MB/s transfers
    da5: Command Queueing enabled
    da5: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da6 at mps0 bus 0 scbus0 target 21 lun 0
    da6: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da6: Serial Number         955AK57IFS9A
    da6: 600.000MB/s transfers
    da6: Command Queueing enabled
    da6: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da11 at mps0 bus 0 scbus0 target 26 lun 0
    da11: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da11: Serial Number         Y5N9K731FS9A
    da11: 600.000MB/s transfers
    da11: Command Queueing enabled
    da11: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da7 at mps0 bus 0 scbus0 target 22 lun 0
    da7: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da7: Serial Number         95GAK5LRFS9A
    da7: 600.000MB/s transfers
    da7: Command Queueing enabled
    da7: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da8 at mps0 bus 0 scbus0 target 23 lun 0
    da8: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da8: Serial Number         Y5NAK6R9FS9A
    da8: 600.000MB/s transfers
    da8: Command Queueing enabled
    da8: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da9 at mps0 bus 0 scbus0 target 24 lun 0
    da9: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da9: Serial Number         Y5REKBRZFS9A
    da9: 600.000MB/s transfers
    da9: Command Queueing enabled
    da9: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da10 at mps0 bus 0 scbus0 target 25 lun 0
    da10: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da10: Serial Number         Y5RDKB6JFS9A
    da10: 600.000MB/s transfers
    da10: Command Queueing enabled
    da10: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)

[root@freenas] ~# zdb | grep ashift
            ashift: 12
            ashift: 12
            ashift: 9
            ashift: 12
            ashift: 12
            ashift: 12
            ashift: 12

It sounds like I need to recreate the pool in order to align them drives correctly. My question is, just how much of a performance hit is this currently costing me? And is there any less invasive way to fix it?

 

[eroji]

Guess this is not a very common issue since no one is chiming in.

 

[cperalt1]

The only way I know is to destroy and create the pool since the ashift value is set at pool create time. What was the ashift value on pool creation? If 12 you might get away with replacing the one with a bad ashift value

 

[eroji]

The only way I know is to destroy and create the pool since the ashift value is set at pool create time. What was the ashift value on pool creation? If 12 you might get away with replacing the one with a bad ashift value

I am not entirely sure how to read that ashift value output as it returns multiple lines. But I assume that it started with 12 but became 9 in the middle means that not all vdevs are incorrectly aligned. Not quite sure how I'd go about fixing that single or multiple vdev though if that is the case.

 

[cperalt1]

ashift 12 = 4k block
ashift 9 = 512 block

How many disks in pool. What kind of pool. How many vdevs in pool.

 

[eroji]

12 disks of same model Toshiba 5TB 7200RPM. They're all in a mirror+stripe configuration in pair of 2 drives each.

 

[cperalt1]

While I don't use FreeBSD I do remember reading something about using GNOP to set ashift values as I believe the kernel will assign the ashift value based on what is reported by the drive.