FreeNAS is great, but one of the pain points is having to boot off USB disks if you don't want to use an entire SSD (or worse NVMe M.2 SSD) as the OS disk when it only needs ~8GB. While looking into converting an existing FreeBSD box to FreeNAS I realized that it's actually quite simple to make FreeNAS work with a partition, but I didn't see anyone else that has done this, so I thought I would give it a try and post for everyone's benefit/amusement.
I did this with an install in ESXi, as a test. I created a VM with three disks, one 8GB and two 128GB. I installed FreeNAS 11.0-RELEASE (the oldest I could find quickly) on the 8GB disk and didn't let the install wizard touch the other two disks. The disks are da0, da1 and da2 respectively below. If you are using real hardware these would be the USB stick and two physical disks (since we obviously want to mirror this, right), with whatever device names they have.
I then followed a basic manual ZFS root install procedure for a mirrored boot pool:
gpart create -s gpt da1
gpart create -s gpt da2
gpart add -b 40 -s 512k -t freebsd-boot da1
gpart add -b 40 -s 512k -t freebsd-boot da2
gpart add -s 8G -t freebsd-zfs -l root0 da1
gpart add -s 8G -t freebsd-zfs -l root1 da2
gpart add -t freebsd-zfs -l jail0 da1
gpart add -t freebsd-zfs -l jail1 da2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2
I then created a jail pool (as an example use case):
zpool create jail mirror /dev/da1p3 /dev/da2p3
umount /jail
zpool export jail
and imported this pool into FreeNAS using the UI. At this point the system is just a standard FreeNAS machine with a single hand-rolled pool.
Now, since FreeNAS only uses the freenas-boot pool it created from this point on, never any raw device names, we can simply move that pool (waiting for the background resilvers to complete) onto a mirror:
zpool attach freenas-boot /dev/da0p2 /dev/da1p2
zpool attach freenas-boot /dev/da0p2 /dev/da2p2
zpool offline freenas-boot /dev/da0p2
zpool detach freenas-boot /dev/da0p2
At this point you can power down the machine and detach da0 (unplug the USB), then start it up again and it should boot from da1. The device names might change at this point depending on your hardware (mine did), but the entire FreeNAS setup should still be there along with the jail pool. The only "rules" that I can see in the code is that the disks must be partition 2, and the boot partition must be 1. The code also has some provisions for partition 3 on the boot disk to be a swap partition.
I then updated to 11.2-U8, which went as expected. This jumps over the grub/BE transition and that was fine. I then also updated to 11.3-U1, which was also fine. I noticed no interesting differences in the dmesg to what I see on real hardware, and no warnings or other messages. I did notice some other quirks, but those look to be bugs in FreeNAS 11.3-U1 and insufficient flushing of the browser cache between updates...
Obviously, the usual caveats hold: this might destroy the world, kill some kittens, eat your lunch, etc. But given how boot environments work, this should continue to work on FreeNAS, and if it didn't there is likely to be sufficient warning in the release notes that they are completely erasing the boot disks. It should be fairly clear to anyone with enough FreeBSD experience how to alter this to suit their own needs, make sure things are 4K aligned, UEFI vs BIOS, etc.
You can also make other partitions on the disks, and use them for other stuff - I'll likely use the rest of OS disks in the FreeBSD server I'm switching to FreeNAS as L2ARC for a big pool. That way if this doesn't work at some point in the future I won't lose any data. Replacing a failed disk in the mirror would require some manual intervention, but nothing too complex if you followed what was going on above (copy GPT from da1 to da2, add boot code, zpool replace p2 and p3).
I did this with an install in ESXi, as a test. I created a VM with three disks, one 8GB and two 128GB. I installed FreeNAS 11.0-RELEASE (the oldest I could find quickly) on the 8GB disk and didn't let the install wizard touch the other two disks. The disks are da0, da1 and da2 respectively below. If you are using real hardware these would be the USB stick and two physical disks (since we obviously want to mirror this, right), with whatever device names they have.
I then followed a basic manual ZFS root install procedure for a mirrored boot pool:
gpart create -s gpt da1
gpart create -s gpt da2
gpart add -b 40 -s 512k -t freebsd-boot da1
gpart add -b 40 -s 512k -t freebsd-boot da2
gpart add -s 8G -t freebsd-zfs -l root0 da1
gpart add -s 8G -t freebsd-zfs -l root1 da2
gpart add -t freebsd-zfs -l jail0 da1
gpart add -t freebsd-zfs -l jail1 da2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2
I then created a jail pool (as an example use case):
zpool create jail mirror /dev/da1p3 /dev/da2p3
umount /jail
zpool export jail
and imported this pool into FreeNAS using the UI. At this point the system is just a standard FreeNAS machine with a single hand-rolled pool.
Now, since FreeNAS only uses the freenas-boot pool it created from this point on, never any raw device names, we can simply move that pool (waiting for the background resilvers to complete) onto a mirror:
zpool attach freenas-boot /dev/da0p2 /dev/da1p2
zpool attach freenas-boot /dev/da0p2 /dev/da2p2
zpool offline freenas-boot /dev/da0p2
zpool detach freenas-boot /dev/da0p2
At this point you can power down the machine and detach da0 (unplug the USB), then start it up again and it should boot from da1. The device names might change at this point depending on your hardware (mine did), but the entire FreeNAS setup should still be there along with the jail pool. The only "rules" that I can see in the code is that the disks must be partition 2, and the boot partition must be 1. The code also has some provisions for partition 3 on the boot disk to be a swap partition.
I then updated to 11.2-U8, which went as expected. This jumps over the grub/BE transition and that was fine. I then also updated to 11.3-U1, which was also fine. I noticed no interesting differences in the dmesg to what I see on real hardware, and no warnings or other messages. I did notice some other quirks, but those look to be bugs in FreeNAS 11.3-U1 and insufficient flushing of the browser cache between updates...
Obviously, the usual caveats hold: this might destroy the world, kill some kittens, eat your lunch, etc. But given how boot environments work, this should continue to work on FreeNAS, and if it didn't there is likely to be sufficient warning in the release notes that they are completely erasing the boot disks. It should be fairly clear to anyone with enough FreeBSD experience how to alter this to suit their own needs, make sure things are 4K aligned, UEFI vs BIOS, etc.
You can also make other partitions on the disks, and use them for other stuff - I'll likely use the rest of OS disks in the FreeBSD server I'm switching to FreeNAS as L2ARC for a big pool. That way if this doesn't work at some point in the future I won't lose any data. Replacing a failed disk in the mirror would require some manual intervention, but nothing too complex if you followed what was going on above (copy GPT from da1 to da2, add boot code, zpool replace p2 and p3).