Help regarding different sector drives (512 vs 4k)

Discussion in 'Solaris, Nexenta, OpenIndiana, and napp-it' started by nle, Jul 17, 2018.

  1. nle

    nle Member

    Joined:
    Oct 24, 2012
    Messages:
    185
    Likes Received:
    6
    Hi, I recently upgraded my zpool2 from (6 x) 2TB drives to HGST 4TB SAS HUS726040ALS214, unfortunately, a couple of the drives I ordered was showing issues, and the seller ran out of them before I could get replacements (he refunded me for the drives).

    Therefore I ordered a couple of HGST Ultrastar 7K6000 HUS726040AL5210 4TB SAS 128MB 12Gb/s 7.2K 3.5". The SN matched so everything should be good (I presumed).

    Unfortunately, when I tried to replace one of the sketchy drives in my pool I get this error:

    Code:
    cannot replace c14t5000CCA25D52E475d0 with c3t5000CCA243014F58d0: devices have different sector alignment
    I've tried to read a little bit about different sector sizes, and it looks like my pool is created with ashift=9, and as far as I can understand that is 512 b sectors.

    Code:
    zdb -C | grep ashift
                ashift: 9
    Any advice on what to do? (and shouldn't two drives with "the same" SN have the same kind of sectors?)

    Thank you.
     
    #1
  2. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,603
    Likes Received:
    1,369
    confirm they actually do have a weird sector size (not 512):

    Code:
    sg_readcap -l /dev/da1
    Then format them back to 512. best to start this in a screen or tmux session as it will take several hours:

    Code:
    sg_format --format --size=512 /dev/da1
    replace da1 with your actual empty drive, of course

    Once finished, some drives require a complete restart (remove power from the disk, then repower it) to come back up properly with the new size
     
    #2
    nle likes this.
  3. nle

    nle Member

    Joined:
    Oct 24, 2012
    Messages:
    185
    Likes Received:
    6
    Thank you, but it seems like I don't have those commands available by default. And I can't seem to find a package to download either. It seems that it should be a package that exists.

    I'm running OmniOS v11 r151026.

    Also, is it smarter to set up a new zpool with ashift 12 (while I still have enough external drive space to move the data)? I get the impression that might be the way to go these days? (Since 4k drives are more common than 512)?

    (Edit: I found sg3_utils here: Package Catalog)
     
    #3
    Last edited: Jul 18, 2018
  4. nle

    nle Member

    Joined:
    Oct 24, 2012
    Messages:
    185
    Likes Received:
    6
    Update. Ended up recreating the pool with ashift=12 for better compatibility down the road. I also followed your advice and reformated the drives with sg_format.
     
    #4
  5. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,338
    Likes Received:
    784
    Ashift is a vdev property, not a pool property, so care about whenever you add a vdev to a pool. The ashift value is automatically based on the physical blocksize of disks. Whenever any disk in a vdev reports 4k physical, the vdev is ashift=12 (4k).

    This is the case when a disk is 4k physical (512e,4kn) or when you force 4k via sd.conf, see ZFS and Advanced Format disks - illumos - illumos wiki
     
    #5
    nle likes this.
  6. nle

    nle Member

    Joined:
    Oct 24, 2012
    Messages:
    185
    Likes Received:
    6
    Thanks for the info @gea. I only have one vdev in this pool.
     
    #6
  7. heliokz

    heliokz New Member

    Joined:
    Aug 24, 2018
    Messages:
    3
    Likes Received:
    0
    Hi guys!
    This is my first post. I'm experiencing the same issue with a WD20EFRX-68E. All the other 3 working drives are:
    2 ST2000VN000
    1 ST2000DM001 (the second one died)

    Code:
    root@napp-it-15d:~# zdb | egrep 'ashift| name'
        name: 'rpool'
                ashift: 9
        name: 'x2mirrorpool'
                ashift: 9
                ashift: 9
    
    Code:
    root@napp-it-15d:~# zpool replace x2mirrorpool c4t5000C5004EB6B30Bd0 c4t50014EE2100DA3F3d0
    cannot replace c4t5000C5004EB6B30Bd0 with c4t50014EE2100DA3F3d0: devices have different sector alignment
    
    As per fohdeesha's advice, I did this:
    Code:
    root@napp-it-15d:~# sg_readcap -l /dev/rdsk/c4t50014EE2100DA3F3d0
    Read Capacity results:
       Protection: prot_en=0, p_type=0, p_i_exponent=0
       Logical block provisioning: lbpme=0, lbprz=0
       Last logical block address=3907029167 (0xe8e088af), Number of logical blocks=3907029168
       Logical block length=512 bytes
       Logical blocks per physical block exponent=3 [so physical block length=4096 bytes]
       Lowest aligned logical block address=0
    Hence:
       Device size: 2000398934016 bytes, 1907729.1 MiB, 2000.40 GB
    I did the same thing with all the 3 others, and the output is exactly the same.
    From this I understand all 4 are 4kn. So what's the problem here?

    By the way, I'm on a HP MicroServer Gen8 under ESXi 6.0u2 with this napp-it (OmniOS v11 r151022) as a VM. The drives are attached to a Perc H310(LSI 9211-8i IT mode) in pci passthrough mode.
     
    #7
  8. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,338
    Likes Received:
    784
    The three Seagate disks are 4K physical that can emulate 512-Byte sectors so they are 512e not 4kn.
    Your pool is ashift=9 (512B) this means that you need a disk that reports 512n for a replacement.

    I would buy a whole new pool (ex 2 x 4TB as mirror) and transfer the data via zfs send
    and use the current disks for backup.

    Only other option is to backup data and to recreate the pool with ashift=12 (4k).
    This would allow a future replacement with 512e/4kn disks.

    Up from newest OmniOS it may be possible to remove an asift=9 vdev amd add it again as asift=12 for a basic or a mirror vdev. Up from Solaris 11.4 this is possible for any vdev type incl Raid-Z(1-3). In all cases the pool is incompatible then with former OS releases.
     
    #8
    Last edited: Aug 24, 2018
  9. heliokz

    heliokz New Member

    Joined:
    Aug 24, 2018
    Messages:
    3
    Likes Received:
    0
    Thanks a lot for your input. I'll go the "recreate the pool with ashift=12 (4k)" route as I'm low on budget.
    Before going that long route, can I try this?:
    Code:
    sg_format --format --size=512 /dev/rdsk/c4t50014EE2100DA3F3d0
    Is there any risk by formatting that way if I'm to follow your other solution?
     
    #9
  10. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,338
    Likes Received:
    784
    Ashift=12 is a vdev property. When you create a vdev the ashift value is set depending on the physical sectorsize value that the disk reports (4k). For a disk with 512B physical sectorsize you can overrule this by a setting in sd.conf.

    A format that was done prior adding to a vdev should be irrelevant as ZFS use its own format.
     
    #10
  11. heliokz

    heliokz New Member

    Joined:
    Aug 24, 2018
    Messages:
    3
    Likes Received:
    0
    Finished recreating the pool with ashift: 12 as sg_format didn't give any result. Thanks for your kind help!
     
    #11
Similar Threads: Help regarding
Forum Title Date
Solaris, Nexenta, OpenIndiana, and napp-it Pool Degraded - Help! May 25, 2019
Solaris, Nexenta, OpenIndiana, and napp-it Help w/ ACL editing on Windows Mar 15, 2018
Solaris, Nexenta, OpenIndiana, and napp-it Help Proofreading an ESXi AIO Jan 7, 2018
Solaris, Nexenta, OpenIndiana, and napp-it NappIt AIO esxi freeze issue ... help requested! Nov 17, 2017
Solaris, Nexenta, OpenIndiana, and napp-it Help... lockups Dec 23, 2016

Share This Page