[SOLVED] zpool SPECIAL DEVICE mirror disk add/remove syntax

[expertaz]

Hello everyone,
Having ZFS pool on proxmox 7.4-17 like below,

Anyway to add 3nd device to SPECIAL DEVICE mirror(3 way mirror)? And after resilver remove one of the old one?

OR
Remove disk form SPECIAL DEVICE mirror and add new one?
Can anyone share SPECIAL DEVICE mirror disk add/remove syntax?


------------------------------------------------------------------------------------------------------------------------------------------------
pool: XPool
state: ONLINE
scan: scrub repaired 0B in 09:12:08 with 0 errors on Sun Feb 11 09:36:20 2024
config:

NAME STATE READ WRITE CKSUM
XPool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-ST14000NM001G-2KJ103 ONLINE 0 0 0
ata-ST14000NM001G-2KJ103 ONLINE 0 0 0
ata-ST14000NM001G-2KJ103 ONLINE 0 0 0
special
mirror-2 ONLINE 0 0 0
ata-KINGSTON_SEDC500M1920G ONLINE 0 0 0
ata-KINGSTON_SEDC500M1920G ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
ata-MARVELL_SD88SA024SA0_SUN24G ONLINE 0 0 0
ata-MARVELL_SD88SA024SA0_SUN24G ONLINE 0 0 0

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Package version:

proxmox-ve: 7.4-1 (running kernel: 5.15.131-2-pve) pve-manager: 7.4-17 (running version: 7.4-17/513c62be) pve-kernel-5.15: 7.4-10 pve-kernel-5.15.136-1-pve: 5.15.136-1 pve-kernel-5.15.131-2-pve: 5.15.131-3 pve-kernel-5.15.102-1-pve: 5.15.102-1 ceph-fuse: 15.2.17-pve1 corosync: 3.1.7-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx4 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve2 libproxmox-acme-perl: 1.4.4 libproxmox-backup-qemu0: 1.3.1-1 libproxmox-rs-perl: 0.2.1 libpve-access-control: 7.4.1 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.4-2 libpve-guest-common-perl: 4.2-4 libpve-http-server-perl: 4.2-3 libpve-rs-perl: 0.7.7 libpve-storage-perl: 7.4-3 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 5.0.2-2 lxcfs: 5.0.3-pve1 novnc-pve: 1.4.0-1 proxmox-backup-client: 2.4.6-1 proxmox-backup-file-restore: 2.4.6-1 proxmox-kernel-helper: 7.4-1 proxmox-mail-forward: 0.1.1-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.7.3 pve-cluster: 7.3-3 pve-container: 4.4-6 pve-docs: 7.4-2 pve-edk2-firmware: 3.20230228-4~bpo11+2 pve-firewall: 4.3-5 pve-firmware: 3.6-6 pve-ha-manager: 3.6.1 pve-i18n: 2.12-1 pve-qemu-kvm: 7.2.0-8 pve-xtermjs: 4.16.0-2 qemu-server: 7.4-4 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.8.0~bpo11+3 vncterm: 1.7-1 zfsutils-linux: 2.1.14-pve1

 

[ekke]

same as ordinry syntax. someone correct me but its like this:

zpool remove xpool ata-KINGSTON_SEDC500M1920G (drive to remove)

zpool attach xpool ata-KINGSTON_SEDC500M1920G (existing drive in already existing mirror) ata-NEWDRIVEHERE

use replace to replace a drive.

zpool-replace.8 — OpenZFS documentation

openzfs.github.io

 

[expertaz]

Ekke thank you for your quick reply,will try .
How about adding 3rd device to SPECIAL DEVICE mirror(3 way mirror)?

 

[ekke]

Ekke thank you for your quick reply,will try .
How about adding 3rd device to SPECIAL DEVICE mirror(3 way mirror)?

should be this one

zpool attach xpool ata-KINGSTON_SEDC500M1920G (existing drive in already existing mirror) ata-NEWDRIVEHERE

 

[expertaz]

update:
1-------------------------------------------------------------------------------------------------------------------------
Made new test pool with SPECIAL DEVICE(SD) 2 disk in mirror

root@pvetest:~# zpool status
pool: tb3
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
tb3 ONLINE 0 0 0
scsi-35000c50041f1a5ef ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11319 ONLINE 0 0 0
ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11311 ONLINE 0 0 0

2-----------------------------------------------------------------------------------------------------------------------------
Added 3 disk to SD mirror with success:

root@pvetest:~# zpool attach tb3 ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11311 ata-KINGSTON_SEDC500M480G_50026B7283069BA3

root@pvetest:~# zpool status
pool: tb3
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Feb 26 14:30:35 2024
45.3G / 45.3G scanned, 1.15G / 45.3G issued at 197M/s
754M resilvered, 2.54% done, 00:03:49 to go
config:

NAME STATE READ WRITE CKSUM
tb3 ONLINE 0 0 0
scsi-35000c50041f1a5ef ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11319 ONLINE 0 0 0
ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11311 ONLINE 0 0 0
ata-KINGSTON_SEDC500M480G_50026B7283069BA3 ONLINE 0 0 0 (resilvering)

3------------------------------------------------------------------------------------------------------------------------------------------------
Last tested SD mirror disk remove after resilver:

root@pvetest:~# zpool iostat -vy 10 1
capacity operations bandwidth
pool alloc free read write read write
------------------------------------------------- ----- ----- ----- ----- ----- -----
tb3 45.3G 3.11T 495 51 7.65M 562K
scsi-35000c50041f1a5ef 44.9G 2.67T 495 3 7.65M 41.2K
special - - - - - -
mirror-1 379M 444G 0 48 0 521K
ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11319 - - 0 16 0 174K
ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11311 - - 0 15 0 174K
ata-KINGSTON_SEDC500M480G_50026B7283069BA3 - - 0 15 0 174K
------------------------------------------------- ----- ----- ----- ----- ----- -----

root@pvetest:~# zpool remove tb3 ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11311
cannot remove ata-SAMSUNG_MZ7L3480HCHQ-00A07_S664NE0RC11311: operation not supported on this type of pool

Any help?

 

[expertaz]

Update2:
when using
detach instead of remove its working,
@ekke thank you for your great support

 

[الحيوانات بالعربي]

thank you