Help regarding different sector drives (512 vs 4k)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nle

Member
Oct 24, 2012
204
11
18
Hi, I recently upgraded my zpool2 from (6 x) 2TB drives to HGST 4TB SAS HUS726040ALS214, unfortunately, a couple of the drives I ordered was showing issues, and the seller ran out of them before I could get replacements (he refunded me for the drives).

Therefore I ordered a couple of HGST Ultrastar 7K6000 HUS726040AL5210 4TB SAS 128MB 12Gb/s 7.2K 3.5". The SN matched so everything should be good (I presumed).

Unfortunately, when I tried to replace one of the sketchy drives in my pool I get this error:

Code:
cannot replace c14t5000CCA25D52E475d0 with c3t5000CCA243014F58d0: devices have different sector alignment
I've tried to read a little bit about different sector sizes, and it looks like my pool is created with ashift=9, and as far as I can understand that is 512 b sectors.

Code:
zdb -C | grep ashift
            ashift: 9
Any advice on what to do? (and shouldn't two drives with "the same" SN have the same kind of sectors?)

Thank you.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,729
3,083
113
33
fohdeesha.com
confirm they actually do have a weird sector size (not 512):

Code:
sg_readcap -l /dev/da1
Then format them back to 512. best to start this in a screen or tmux session as it will take several hours:

Code:
sg_format --format --size=512 /dev/da1
replace da1 with your actual empty drive, of course

Once finished, some drives require a complete restart (remove power from the disk, then repower it) to come back up properly with the new size
 
  • Like
Reactions: nle

nle

Member
Oct 24, 2012
204
11
18
Thank you, but it seems like I don't have those commands available by default. And I can't seem to find a package to download either. It seems that it should be a package that exists.

I'm running OmniOS v11 r151026.

Also, is it smarter to set up a new zpool with ashift 12 (while I still have enough external drive space to move the data)? I get the impression that might be the way to go these days? (Since 4k drives are more common than 512)?

(Edit: I found sg3_utils here: Package Catalog)
 
Last edited:

nle

Member
Oct 24, 2012
204
11
18
Update. Ended up recreating the pool with ashift=12 for better compatibility down the road. I also followed your advice and reformated the drives with sg_format.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,184
113
DE
Ashift is a vdev property, not a pool property, so care about whenever you add a vdev to a pool. The ashift value is automatically based on the physical blocksize of disks. Whenever any disk in a vdev reports 4k physical, the vdev is ashift=12 (4k).

This is the case when a disk is 4k physical (512e,4kn) or when you force 4k via sd.conf, see ZFS and Advanced Format disks - illumos - illumos wiki
 
  • Like
Reactions: nle

heliokz

New Member
Aug 24, 2018
3
0
1
Hi guys!
This is my first post. I'm experiencing the same issue with a WD20EFRX-68E. All the other 3 working drives are:
2 ST2000VN000
1 ST2000DM001 (the second one died)

Code:
root@napp-it-15d:~# zdb | egrep 'ashift| name'
    name: 'rpool'
            ashift: 9
    name: 'x2mirrorpool'
            ashift: 9
            ashift: 9
Code:
root@napp-it-15d:~# zpool replace x2mirrorpool c4t5000C5004EB6B30Bd0 c4t50014EE2100DA3F3d0
cannot replace c4t5000C5004EB6B30Bd0 with c4t50014EE2100DA3F3d0: devices have different sector alignment
As per fohdeesha's advice, I did this:
Code:
root@napp-it-15d:~# sg_readcap -l /dev/rdsk/c4t50014EE2100DA3F3d0
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last logical block address=3907029167 (0xe8e088af), Number of logical blocks=3907029168
   Logical block length=512 bytes
   Logical blocks per physical block exponent=3 [so physical block length=4096 bytes]
   Lowest aligned logical block address=0
Hence:
   Device size: 2000398934016 bytes, 1907729.1 MiB, 2000.40 GB
I did the same thing with all the 3 others, and the output is exactly the same.
From this I understand all 4 are 4kn. So what's the problem here?

By the way, I'm on a HP MicroServer Gen8 under ESXi 6.0u2 with this napp-it (OmniOS v11 r151022) as a VM. The drives are attached to a Perc H310(LSI 9211-8i IT mode) in pci passthrough mode.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,184
113
DE
The three Seagate disks are 4K physical that can emulate 512-Byte sectors so they are 512e not 4kn.
Your pool is ashift=9 (512B) this means that you need a disk that reports 512n for a replacement.

I would buy a whole new pool (ex 2 x 4TB as mirror) and transfer the data via zfs send
and use the current disks for backup.

Only other option is to backup data and to recreate the pool with ashift=12 (4k).
This would allow a future replacement with 512e/4kn disks.

Up from newest OmniOS it may be possible to remove an asift=9 vdev amd add it again as asift=12 for a basic or a mirror vdev. Up from Solaris 11.4 this is possible for any vdev type incl Raid-Z(1-3). In all cases the pool is incompatible then with former OS releases.
 
Last edited:

heliokz

New Member
Aug 24, 2018
3
0
1
Thanks a lot for your input. I'll go the "recreate the pool with ashift=12 (4k)" route as I'm low on budget.
Before going that long route, can I try this?:
Code:
sg_format --format --size=512 /dev/rdsk/c4t50014EE2100DA3F3d0
Is there any risk by formatting that way if I'm to follow your other solution?
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,184
113
DE
Ashift=12 is a vdev property. When you create a vdev the ashift value is set depending on the physical sectorsize value that the disk reports (4k). For a disk with 512B physical sectorsize you can overrule this by a setting in sd.conf.

A format that was done prior adding to a vdev should be irrelevant as ZFS use its own format.
 

heliokz

New Member
Aug 24, 2018
3
0
1
Finished recreating the pool with ashift: 12 as sg_format didn't give any result. Thanks for your kind help!