How to build a OpenSolaris derived ZFS Storage Server
- Thread starter gea
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Note this howto is about 5 years old now. If I were building an illumos-based storage server today, I'd be more likely to use as a distribution one of:
- OmniOS - Great lightweight general purpose UNIX OS. Installs to disk.
- SmartOS - Next generation hypervisor OS, boots from ephemeral source (PXE, USB stick, CD, etc) and sets all storage aside into a zfs pool for guest zones.
Time to update a few general items for a Solarish based ZFS server
Oracle Solaris 11.4
This is the enterprise related ZFS option with native ZFS and a support option up to 2034+
https://blogs.oracle.com/solaris/long-live-solaris-11-until-at-least-2034-to-be-exact-v2
Up to this year, Oracle Solaris was the most feature rich and the fastest ZFS server in my tests. It comes with native ZFS that supports encryption, sequential fast resilvering, vdev remove (unlike Open-ZFS with support for all vdevs incl Raid-Z), dedup2, SMB 3.1 and NFS 4.1 among other features.
This year 2019 was the changing point feature wise. Open-ZFS now includes encryption, fast sequential (sorted) resilvering, special vdevs, vdev remove (basic and mirror only,) system checkpoints, trim, force ashift and many other features. They are now all in Illumos, the free Solaris fork and therefor in OpenIndiana (that is ongoing Illumos with a huge repository that includes desktop apps) and OmniOS, a storage related distribution with a freeze of Illumos in a stable and long term stable and a commercial support option with often be-weekly security and bugfix updates - perfect for a robust and stable production storage server.
Current version of OmniOS is 151032 with all the new Open-ZFS features, SMB 3.02, many updates related to hardware and Comstar iSCSI
see
Release notes: omniosorg/omnios-build
Issue tracker: Issues - illumos
Discuss: Topicbox
The above enterprise ready Unix operating systems are managed via CLI or you can install a desktop environment (Solaris and OpenIndiana). As an add on, you can use my napp-it web-ui tool to manage all ZFS related items (pools, (special) vdevs, zvols, filesystems, snaps, clones, trim, raid management, share management, enclosure management incl a disk map, smart, remote replication, acl management, realtime monitoring, acceleration with background agents, ZFS clustering etc). You can update/bugfix the OS independently from napp-it updates. There is no lock in between them (or a special hardware environment).
napp-it manuals: napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris : Manual
New feature in current 19.dev: pool status with all relevant informations:
I will update this from time to time.
Oracle Solaris 11.4
This is the enterprise related ZFS option with native ZFS and a support option up to 2034+
https://blogs.oracle.com/solaris/long-live-solaris-11-until-at-least-2034-to-be-exact-v2
Up to this year, Oracle Solaris was the most feature rich and the fastest ZFS server in my tests. It comes with native ZFS that supports encryption, sequential fast resilvering, vdev remove (unlike Open-ZFS with support for all vdevs incl Raid-Z), dedup2, SMB 3.1 and NFS 4.1 among other features.
This year 2019 was the changing point feature wise. Open-ZFS now includes encryption, fast sequential (sorted) resilvering, special vdevs, vdev remove (basic and mirror only,) system checkpoints, trim, force ashift and many other features. They are now all in Illumos, the free Solaris fork and therefor in OpenIndiana (that is ongoing Illumos with a huge repository that includes desktop apps) and OmniOS, a storage related distribution with a freeze of Illumos in a stable and long term stable and a commercial support option with often be-weekly security and bugfix updates - perfect for a robust and stable production storage server.
Current version of OmniOS is 151032 with all the new Open-ZFS features, SMB 3.02, many updates related to hardware and Comstar iSCSI
see
Release notes: omniosorg/omnios-build
Issue tracker: Issues - illumos
Discuss: Topicbox
The above enterprise ready Unix operating systems are managed via CLI or you can install a desktop environment (Solaris and OpenIndiana). As an add on, you can use my napp-it web-ui tool to manage all ZFS related items (pools, (special) vdevs, zvols, filesystems, snaps, clones, trim, raid management, share management, enclosure management incl a disk map, smart, remote replication, acl management, realtime monitoring, acceleration with background agents, ZFS clustering etc). You can update/bugfix the OS independently from napp-it updates. There is no lock in between them (or a special hardware environment).
napp-it manuals: napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris : Manual
New feature in current 19.dev: pool status with all relevant informations:
I will update this from time to time.
napp-it Version 20.x
I have uploaded a pre version of next napp-it v19.12 noncommercial homeuse
and napp-it 20.01 pro to support the newest features of Oracle Solaris and expecially OmniOS/OpenIndiana.
- ZFS encryption with web/filebased keys, a http/https keyserver with an HA option,
keysplit for two locations, automount after reboot and user lock/unlock via SMB
http://napp-it.org/doc/downloads/zfs_encryption.pdf
- special vdevs
https://www.napp-it.org/doc/downloads/special-vdev.pdf
- trim
- force ashift when adding a vdev
- protection against accidentially adding a basic vdev to a pool
more (all features from 19.dev)
napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris Changelog
I have uploaded a pre version of next napp-it v19.12 noncommercial homeuse
and napp-it 20.01 pro to support the newest features of Oracle Solaris and expecially OmniOS/OpenIndiana.
- ZFS encryption with web/filebased keys, a http/https keyserver with an HA option,
keysplit for two locations, automount after reboot and user lock/unlock via SMB
http://napp-it.org/doc/downloads/zfs_encryption.pdf
- special vdevs
https://www.napp-it.org/doc/downloads/special-vdev.pdf
- trim
- force ashift when adding a vdev
- protection against accidentially adding a basic vdev to a pool
more (all features from 19.dev)
napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris Changelog
I have got first mails about the new v20 pre.
The new version does not only checks disk smart by smart tests (ok/failed) but also analyze several smart values that may predict a future failure. Up from a certain value I display a smart warning in disk/smart overview.
Are the disks then bad: No
But in a critical production system, I would replace. At home/lab I would propably ignore and wait until a disk fails really.
Background Smart checks can be disabled in Services > ACC
What SMART Stats Tell Us About Hard Drives
The new version does not only checks disk smart by smart tests (ok/failed) but also analyze several smart values that may predict a future failure. Up from a certain value I display a smart warning in disk/smart overview.
Are the disks then bad: No
But in a critical production system, I would replace. At home/lab I would propably ignore and wait until a disk fails really.
Background Smart checks can be disabled in Services > ACC
What SMART Stats Tell Us About Hard Drives
Last edited:
Ok, an update about questions I often get about OpenSolaris derived ZFS server distributions.
If you want OSX or Windows,
you select OSX 11+ or Windows 10+ as your only concern are bugfix and security updates.
Handling and software are quite identical between releases.
If you want Linux,
you select a distribution under support ex Red Hat, Debian or Ubuntu.
Handling and software repositories differ. Your use case and preferences decide the choice.
If you want a Solaris based Unix,
situation is similar to Linux, so which Solaris flavour for a ZFS Unix server?
History
Over 12 years ago, Oracle aquired Sun with its server hardware, sophisticated technologies like ZFS, dtrace, Linux and Solaris container based virtualization (zones, similar to Docker years later), Java and databases like MySQL together with Solaris a leading Unix that was then quite the newest Unix, often years ahead of others and one that set milestones. One of the first Oracle actions was to complete (and end) Sun OpenSolaris beta to the current closed source commercial Oracle Solaris 11 stable with a support guarantee at least until 2034.
Luckily Sun OpenSolaris beta was OpenSource.
This is why we now have a free ZFS with encryption, dtrace, LX Linux container and the free Solaris forks based on Illumos that continued development of OpenSolaris and ZFS. As native ZFS was then closed source, Illumos introduced a different versioning method (ZFS v5000 with features) to allow newer ZFS options independently from Oracle ZFS versions, Managing ZFS File Systems in Oracle Solaris 11.4
This is why you can now select between different options and distributions for a Solaris based Unix ZFS server
Oracle Solaris 11.4 with native ZFS
docs.oracle.com
This is the datacenter option due the extreme long support guarantee. In my tests Solaris was often the fastest server regarding ZFS pools or SMB performance with for many years superior features over Open-ZFS. For commercial use a support contract (min 1k$/year per server) is mandatory that gives access to newest ZFS versions with support, bug and security fixes.
For noncommercial developper and home use, there is the free to download Solaris 11.4 CBE. This is the ongoing beta of the current Solaris 11.4 development and gives also access to newest ZFS versions and Solaris fixes, https://forums.servethehome.com/ind...ee-for-personal-and-non-commercial-use.36197/
Use Solaris when you want the best performance. Commercial driver support ex for IB adapters or newest SMB 3.1.1 feature support are also a reason for Solaris as long as you do not need Open-ZFS compatibility. You can use Solaris as a minimalistic ultra stable server or add a Gnome3 desktop, Oracle Solaris 11.4 Desktop: An Overview - Oracle® Solaris 11.4 Desktop
Illumos
Illumos is the Opensource fork of OpenSolaris and the roof under which many firms coordinate their common developments around the Solaris fork. It provides the same packaging system IPS, the kernelbased NFS and SMB server with ntfs alike ACL support and Comstar, the enterprise FC/iSCSI stack quite comparable to the Solaris ones but with a compatible Open-ZFS v5000 with features instead native ZFS versions. Newer Open-ZFS features are included in Illumos after an additional test and include process. You can see Illumos like Linux as the common development platform for OpenSolaris derived Unix distributions mainly for X86 but partly for Sparc or ARM, illumos or Topicbox
OpenIndiana (Illumos distribution)
This is more or less the successor of Sun OpenSolaris as it is also provided in a server and a desktop edition (with a Mate desktop, browser, mail and office apps). OpenIndiana is not only one of the Illumos distributions but a defacto Illumos reference installation. While there are annual snapshots ex current 2022.10 available as installer isos, a pkg update gives you always the newest Illumos state.
Use OpenIndiana if you want an optional desktop edition with newest ongoing Illumos features. While Illumos in general is very stable there is no OpenIndiane stable or LTS. The OpenIndiana repository includes many apps not available on Solaris or OnniOS. Some say OpenIndiana is similar to Ubuntu on Linux, openindiana – Community-driven illumos Distribution
Dilos and Tribblix (Illumos distributions)
These are Illumos distribution with different packaging methods SVR4 or a Debian alike apt or available in a Sparc edition (Sun server), About DilOS - DilOS and Tribblix
SmartOS Joyent/MNX (Illumos distribution)
This is a specialized commercial but OpenSource Illumos distribution for virtualisation as an alternative to ESXi and ProxMox. While I am very keen of the idea of a minimalistic system on a small USB bootstick with everything relevant on a ZFS datapool, SmartOS is quite limited in global zone options so not as optimal for a basic general use filer, SmartOS or SmartOS - Wikipedia
NexentaStor/DDN (Illumos distribution)
This is a commercial storage server option, Home
OmniOS (Illumos distribution)
OmniOS is the OpenSource alternative to Solaris if you want an enterprise class ZFS Unix server with a beta, stable, lts and often biweekly bug and security fixes. Like Solaris there is a commercial support option with help from OS devs.
With dedicated repositories per stable/LTS release OmniOS offers stability and robustness. While a pkg update or apt update on other platforms updates all packages to newest versions (often with newer and untested or removed features), OmniOS freezes packages and only offers bug and security updates with a pkg update. To upgrade to a newer OmniOS version you must switch repository first to point to the newer version. This and the smooth and well tested upgrade process with the option to go back to a formar OS release via bootenvironments is the reason why OmniOS is my preferred option. Some say the OmniOS distribution is like Debian on Linux, OmniOS Community Edition
Just to add
While Solaris, OmniOS or OpenIndiana are quite feature complete and self sufficient 64bit operating systems for a storage server as Sun had added what is needed as OS and kernel services, they share IPS as the system to install or update OS and applications. Do not cross use repositories as there are distribution related dependencies. If the native repository does not offer an application and you do not want to compile yourself, use the sfe repositories for your selected OS or use pkgin for Illumos, napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris : Downloads
News, tips and tricks for a Solaris based server,
more Manuals around napp-it
www.napp-it.org
If you want OSX or Windows,
you select OSX 11+ or Windows 10+ as your only concern are bugfix and security updates.
Handling and software are quite identical between releases.
If you want Linux,
you select a distribution under support ex Red Hat, Debian or Ubuntu.
Handling and software repositories differ. Your use case and preferences decide the choice.
If you want a Solaris based Unix,
situation is similar to Linux, so which Solaris flavour for a ZFS Unix server?
History
Over 12 years ago, Oracle aquired Sun with its server hardware, sophisticated technologies like ZFS, dtrace, Linux and Solaris container based virtualization (zones, similar to Docker years later), Java and databases like MySQL together with Solaris a leading Unix that was then quite the newest Unix, often years ahead of others and one that set milestones. One of the first Oracle actions was to complete (and end) Sun OpenSolaris beta to the current closed source commercial Oracle Solaris 11 stable with a support guarantee at least until 2034.
Luckily Sun OpenSolaris beta was OpenSource.
This is why we now have a free ZFS with encryption, dtrace, LX Linux container and the free Solaris forks based on Illumos that continued development of OpenSolaris and ZFS. As native ZFS was then closed source, Illumos introduced a different versioning method (ZFS v5000 with features) to allow newer ZFS options independently from Oracle ZFS versions, Managing ZFS File Systems in Oracle Solaris 11.4
This is why you can now select between different options and distributions for a Solaris based Unix ZFS server
Oracle Solaris 11.4 with native ZFS
Oracle Solaris 11 Documentation - Oracle Solaris Documentation
This is the datacenter option due the extreme long support guarantee. In my tests Solaris was often the fastest server regarding ZFS pools or SMB performance with for many years superior features over Open-ZFS. For commercial use a support contract (min 1k$/year per server) is mandatory that gives access to newest ZFS versions with support, bug and security fixes.
For noncommercial developper and home use, there is the free to download Solaris 11.4 CBE. This is the ongoing beta of the current Solaris 11.4 development and gives also access to newest ZFS versions and Solaris fixes, https://forums.servethehome.com/ind...ee-for-personal-and-non-commercial-use.36197/
Use Solaris when you want the best performance. Commercial driver support ex for IB adapters or newest SMB 3.1.1 feature support are also a reason for Solaris as long as you do not need Open-ZFS compatibility. You can use Solaris as a minimalistic ultra stable server or add a Gnome3 desktop, Oracle Solaris 11.4 Desktop: An Overview - Oracle® Solaris 11.4 Desktop
Illumos
Illumos is the Opensource fork of OpenSolaris and the roof under which many firms coordinate their common developments around the Solaris fork. It provides the same packaging system IPS, the kernelbased NFS and SMB server with ntfs alike ACL support and Comstar, the enterprise FC/iSCSI stack quite comparable to the Solaris ones but with a compatible Open-ZFS v5000 with features instead native ZFS versions. Newer Open-ZFS features are included in Illumos after an additional test and include process. You can see Illumos like Linux as the common development platform for OpenSolaris derived Unix distributions mainly for X86 but partly for Sparc or ARM, illumos or Topicbox
OpenIndiana (Illumos distribution)
This is more or less the successor of Sun OpenSolaris as it is also provided in a server and a desktop edition (with a Mate desktop, browser, mail and office apps). OpenIndiana is not only one of the Illumos distributions but a defacto Illumos reference installation. While there are annual snapshots ex current 2022.10 available as installer isos, a pkg update gives you always the newest Illumos state.
Use OpenIndiana if you want an optional desktop edition with newest ongoing Illumos features. While Illumos in general is very stable there is no OpenIndiane stable or LTS. The OpenIndiana repository includes many apps not available on Solaris or OnniOS. Some say OpenIndiana is similar to Ubuntu on Linux, openindiana – Community-driven illumos Distribution
Dilos and Tribblix (Illumos distributions)
These are Illumos distribution with different packaging methods SVR4 or a Debian alike apt or available in a Sparc edition (Sun server), About DilOS - DilOS and Tribblix
SmartOS Joyent/MNX (Illumos distribution)
This is a specialized commercial but OpenSource Illumos distribution for virtualisation as an alternative to ESXi and ProxMox. While I am very keen of the idea of a minimalistic system on a small USB bootstick with everything relevant on a ZFS datapool, SmartOS is quite limited in global zone options so not as optimal for a basic general use filer, SmartOS or SmartOS - Wikipedia
NexentaStor/DDN (Illumos distribution)
This is a commercial storage server option, Home
OmniOS (Illumos distribution)
OmniOS is the OpenSource alternative to Solaris if you want an enterprise class ZFS Unix server with a beta, stable, lts and often biweekly bug and security fixes. Like Solaris there is a commercial support option with help from OS devs.
With dedicated repositories per stable/LTS release OmniOS offers stability and robustness. While a pkg update or apt update on other platforms updates all packages to newest versions (often with newer and untested or removed features), OmniOS freezes packages and only offers bug and security updates with a pkg update. To upgrade to a newer OmniOS version you must switch repository first to point to the newer version. This and the smooth and well tested upgrade process with the option to go back to a formar OS release via bootenvironments is the reason why OmniOS is my preferred option. Some say the OmniOS distribution is like Debian on Linux, OmniOS Community Edition
Just to add
While Solaris, OmniOS or OpenIndiana are quite feature complete and self sufficient 64bit operating systems for a storage server as Sun had added what is needed as OS and kernel services, they share IPS as the system to install or update OS and applications. Do not cross use repositories as there are distribution related dependencies. If the native repository does not offer an application and you do not want to compile yourself, use the sfe repositories for your selected OS or use pkgin for Illumos, napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris : Downloads
News, tips and tricks for a Solaris based server,
more Manuals around napp-it
napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris : Manual
napp-it.org - ZFS NAS SAN Server with user editable web-gui // All In One server = virtualized ready to run ZFS-Server
Last edited: