Force existing root dataset metadata onto added ZFS special vdev without pool destruction?

[aphirst]

System/Hardware/Layout:

• OS: Proxmox
• Pool: gimli (4×16TB 3.5" SATA CMR HDDs, RAIDZ1)
• Current usage: ~21TB used, ~21TB free (mostly large media files + metadata)
• Planned: Add 2×SSD partitions as mirrored special vdev (sized appropriately)
• Available temp space: 8×2TB 2.5" SAS drives (can build ~12TB RAIDZ2 scratch pool, insufficient for full 21TB copy)

Goal:
Keep the 4 SATA HDDs spun down 24/7 (by means of hd-idle) unless genuinely reading file data. Directory listings, find, stat, zfs list, and any metadata probes must be served entirely from the SSD special vdev. Spin-ups are "expensive" (10min idle timeout) and must be avoided for metadata access.

Proposed "Dangling Root" Method:

1. Add special vdev, set special_small_blocks=16K (or 32K, that's a separate problem for me to solve myself)
2. Create new dataset gimli/main (post-special-vdev addition) and make sure the special_small_blocks setting is correctly inherited
3. In turn, zfs send | receive each child dataset (loki/anime, odin/films, etc.) via the 12TB scratch pool into gimli/main/..., destroying originals after each transfer is completed
4. Set mountpoint=none on the original gimli root, mount gimli/main at /gimli

The Problem:
The original pool root dataset (gimli) is "empty" but retains ~140KB of (what I understand to be) directory ZAP objects allocated on the RAIDZ1 vdevs. While mountpoint=none renders it inaccessible to users, the data remains physically on the HDDs. I'm well aware that I'm being paranoid, but this feels "incomplete" - the HDDs still hold pool metadata that could theoretically be accessed if someone manually mounted the root or if ZFS internally references it somehow.

The Question:
Is there any mechanism to force the existing root dataset's dnodes/ZAP blocks (I hope I have the terminology right) to relocate onto the special vdev without destroying the pool?

Specifically:

• Can the root dataset's indirect blocks be rewritten in-place to the special vdev (similar to how special_small_blocks works for new writes, but retroactively)?
• Is there a zpool or zfs command (even undocumented) that defragments/remaps a specific object's block pointers?
• Or is the "unmounted root" method truly the only path when lacking 21TB of external temp space to destroy/recreate the pool?

I want to verify that ls /gimli (which hits the new gimli/main root) will never trigger reads on the underlying HDDs, and that no ZFS administrative command (zfs list, zpool status, zdb, etc.) will touch the HDDs in order to access the old root's ZAP data.

Constraints:

• Cannot destroy and re-create the pool fresh (no 21TB temp storage available)
• Must be verifiable proof that the root directory actually lives on the SSDs (e.g., zdb output showing the data resides on the special vdev, not the RAIDZ1 disks)

Thanks in advance for any insights on whether that 140KB can be forced onto the SSDs, or confirmation on whether I should accept the unmounted old root as "good enough" for spin-down purposes.

 

[etorix]

In principle, zfs rewrite could do it in place, without the need for an intermediate pool but with the same question mark about residual root objects.
If you cannot have a large enough backup pool for a full backup-destroy-restore, I'd suggest to dodge the rabbit hole and take the "good enough" road.

Add special vdev, set special_small_blocks=16K (or 32K, that's a separate problem for me to solve myself)

Guidance here:

Special VDEV (sVDEV) Planning, Sizing, and Considerations

This resource was originally sourced from a Suggestion thread here: https://forums.truenas.com/t/suggestion-better-svdev-planning-oversight-tools-in-gui/394/ Why consider a sVDEV? A sVDEV has the potential to significantly speed up the operations of a spinning hard-drive (HDD) pool. One of...

forums.truenas.com

 

[aphirst]

In principle, zfs rewrite could do it in place, without the need for an intermediate pool but with the same question mark about residual root objects.

Ah, right, zfs rewrite is recently "a thing". I was, however, under the impression it was more for the level of individual files than to apply to the level of entire datasets or pools, so I'm not sure whether it alone addresses the bits of metadata I'm being most paranoid about...