Switching from SmartOS to TrueNAS

knorrhane

New Member
Dec 17, 2016
8
0
1
42
Hi,

I'm currently running SmartOS with a couple of VMs and I'm thinking about trying TrueNAS CORE instead and I would appreciate your input on a couple of things.

- I can't find information about which ZFS version currently being used in TrueNAS as it needs to be the same or higher to be able to import my current zpool. I don't have my server online at the moment and don't remember what ZFS version I'm running but if anyone knows what TrueNAS uses I can check later.

- Is making Linux and Windows VMs in TrueNAS straightforward? I like the lx-zones in SmartOS but they are not keeping up with the latest Linux-releases.

- Anyone has any experience with performance on the ZFS pool using SMB on SmartOS vs TrueNAS? I don't expect any difference but interesting to know if anyone has experience.

That's it, thanks guys!
 

PD_ZFS-User

New Member
Jul 13, 2018
25
10
3
knorrhane,

Gea is an expert on ZFS who frequents these STH forums, but here's some basic information below about feature flags which is the system the zfs developers switched to from the earlier zfs pool versioning. I don't have any experience with SmartOS, but IIRC the last pool version was V28 in OmniOS before the switch to feature flags. Hope this helps with one of your questions.

From ZFS Feature Flags in TrueNAS

OpenZFS' distributed development led to the introduction of Feature Flags. Instead of incrementing version numbers, support for OpenZFS features is indicated by Feature Flags.

Feature Flag states

Feature flags exist in one of three states:

disabled
The Feature Flag is not used by the pool. The pool can be imported on systems that do not support this feature flag.

enabled
The feature has been enabled for use in this pool, but no changes are in effect. The pool can be imported on systems that do not support this feature flag.

active
The on-disk format of the pool includes the changes needed for this feature. Some features may allow for the pool to be imported read-only, while others make the pool completely incompatible with systems that do not support the Feature Flag in question.

Note that many ZFS features, such as compressed ARC or sequential scrub/resilver, do not require on-disk format changes. They do not introduce feature flags and pools used with these features are compatible with systems lacking them.


Overview of commands

To see the Feature Flags supported by the version of ZFS you're running, use man zpool-features.
To view the status of Feature Flags on a pool, use zpool get all poolname | grep feature.
To view available Feature Flags, use zpool upgrade. Feature Flags can be enabled using zpool upgrade poolname.
Feature flags can be selectively enabled at import time with zpool import -o feature@feature_name=enabled poolname. To enable multiple features at once, specify -o feature@feature1=enabled -o feature@feature2=enabled ... for each feature.
 

gea

Well-Known Member
Dec 31, 2010
2,794
963
113
DE
As I see it

The Solaris fork Illumos (OmniOS, OpenIndiana, SmartOS), Free-BSD and Linux are quite comparable regarding ZFS features or performance. Usually you can move pools between them via a simple pool import.

SmartOS is strong on virtualisation and a competitor to ESXi or ProxMox. FreeNAS/TrueNas is more a general use ZFS filer with a web-ui and some virtualisation options. If you are coming from SmartOS, you may look at OmniOS as it has a similar feature set as FreeNAS/ TrueNas with most of SmartOS virtualisation options.

www.omnios.org (minimalistic enterprise class OS)
napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris : Manual (web-ui)

virtualisation options, mostly ported from SmartOS including newer LX images from OmniOS
https://forums.servethehome.com/index.php?threads/virtualisation-options-with-omnios.34296/

Main advantage of Solarish based operating systems:
the lowest overall resource needs to run ZFS (native ZFS environment), especially the Solaris/OpenIndiana text editions and minimalistic OmniOS

Stable/long term stable with rapid bug and security fixes
(Solaris + maintenance, OmniOS /either free or with a commercial support contract)

All major services are from Sun/Oracle and part even of a minimalistic distribution like NFS, SMB or FC/iSCSI. No third party apps needed, perfectly integrated in ZFS.

Especially the Solaris SMB server is unique. It offers best of all Windows compatibility when it comes to permissions (Windows ntfs alike ACL, local SMB groups, Windows sid as extended ZFS attributes, out of the box integration of ZFS snaps as Windows previous versions = it just works). Additionally the multithreaded Solarish SMB server is often faster than SAMBA that is used on Free-BSD or Linux or an option on Illumos.

If you want to use VMs, the fastest option for Windows with the best VM separation is ESXi + OmniOS as a ZFS storage VM, https://napp-it.org/doc/downloads/napp-in-one.pdf.
 
Last edited:
  • Like
Reactions: ecosse

knorrhane

New Member
Dec 17, 2016
8
0
1
42
knorrhane,

Gea is an expert on ZFS who frequents these STH forums, but here's some basic information below about feature flags which is the system the zfs developers switched to from the earlier zfs pool versioning. I don't have any experience with SmartOS, but IIRC the last pool version was V28 in OmniOS before the switch to feature flags. Hope this helps with one of your questions.

From ZFS Feature Flags in TrueNAS

OpenZFS' distributed development led to the introduction of Feature Flags. Instead of incrementing version numbers, support for OpenZFS features is indicated by Feature Flags.

Feature Flag states

Feature flags exist in one of three states:

disabled
The Feature Flag is not used by the pool. The pool can be imported on systems that do not support this feature flag.

enabled
The feature has been enabled for use in this pool, but no changes are in effect. The pool can be imported on systems that do not support this feature flag.

active
The on-disk format of the pool includes the changes needed for this feature. Some features may allow for the pool to be imported read-only, while others make the pool completely incompatible with systems that do not support the Feature Flag in question.

Note that many ZFS features, such as compressed ARC or sequential scrub/resilver, do not require on-disk format changes. They do not introduce feature flags and pools used with these features are compatible with systems lacking them.


Overview of commands

To see the Feature Flags supported by the version of ZFS you're running, use man zpool-features.
To view the status of Feature Flags on a pool, use zpool get all poolname | grep feature.
To view available Feature Flags, use zpool upgrade. Feature Flags can be enabled using zpool upgrade poolname.
Feature flags can be selectively enabled at import time with zpool import -o feature@feature_name=enabled poolname. To enable multiple features at once, specify -o feature@feature1=enabled -o feature@feature2=enabled ... for each feature.
Thanks a lot! I couldn't find that information myself, appreciate it! You are right the pool versioning changed to feature flags a while back, it's been so long since I set up the zpool I have now I forgot all about it. If I remember correctly I have version 28 so should be ok. Thanks again!
 

knorrhane

New Member
Dec 17, 2016
8
0
1
42
As I see it

The Solaris fork Illumos (OmniOS, OpenIndiana, SmartOS), Free-BSD and Linux are quite comparable regarding ZFS features or performance. Usually you can move pools between them via a simple pool import.

SmartOS is strong on virtualisation and a competitor to ESXi or ProxMox. FreeNAS/TrueNas is more a general use ZFS filer with a web-ui and some virtualisation options. If you are coming from SmartOS, you may look at OmniOS as it has a similar feature set as FreeNAS/ TrueNas with most of SmartOS virtualisation options.

www.omnios.org (minimalistic enterprise class OS)
napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris : Manual (web-ui)

virtualisation options, mostly ported from SmartOS including newer LX images from OmniOS
https://forums.servethehome.com/index.php?threads/virtualisation-options-with-omnios.34296/

Main advantage of Solarish based operating systems:
the lowest overall resource needs to run ZFS (native ZFS environment), especially the Solaris/OpenIndiana text editions and minimalistic OmniOS

Stable/long term stable with rapid bug and security fixes
(Solaris + maintenance, OmniOS /either free or with a commercial support contract)

All major services are from Sun/Oracle and part even of a minimalistic distribution like NFS, SMB or FC/iSCSI. No third party apps needed, perfectly integrated in ZFS.

Especially the Solaris SMB server is unique. It offers best of all Windows compatibility when it comes to permissions (Windows ntfs alike ACL, local SMB groups, Windows sid as extended ZFS attributes, out of the box integration of ZFS snaps as Windows previous versions = it just works). Additionally the multithreaded Solarish SMB server is often faster than SAMBA that is used on Free-BSD or Linux or an option on Illumos.

If you want to use VMs, the fastest option for Windows with the best VM separation is ESXi + OmniOS as a ZFS storage VM, https://napp-it.org/doc/downloads/napp-in-one.pdf.
Thanks for the fast reply!

I did run ESXi + OmniOS/napp-it a couple of years ago but started using SmartOS instead to get native ZFS support on the hypervisor as ESXi seemed like an unnecessary overhead for me. I also had problems with ESXi every time I updated it on my hardware so I didn't like that.

I was thinking about running docker containers on TrueNAS as the VMs I have today mostly are different web servers, home assistant, plex etc. which could all be run in docker containers but since TrueNAS CORE is based on FreeBSD I would have to run a Linux VM and then run docker containers. I run a Windows VM which I use for other stuff but I'm not sure I need it going forward.

Interesting what you said about the SMB server, I have 10 Gbe to my workstation from the server and use SMB for file sharing so that is relevant.

Really appreciate your input and it's giving me something to think about what could fit what I want to do! Right now I'm thinking I might try napp-it again as the Web UI is nice or stick with SmartOS. We'll see!
 

EngChiSTH

Member
Jun 27, 2018
95
39
18
Chicago
Have you already looked at Proxmox? Debian based hypervisor with support for zfs (if that is what you want) which you can run whichever way you desire.

In my mind - if you focus on virtualization , go with hypervisor . if your focus is storage, go with something like truenas. Can Truenas do light virtualization in a pitch - yes, sure. is it same feature equivalent as say proxmox, not in my experience.

I use both. they are differently focused tools.
 

knorrhane

New Member
Dec 17, 2016
8
0
1
42
Have you already looked at Proxmox? Debian based hypervisor with support for zfs (if that is what you want) which you can run whichever way you desire.

In my mind - if you focus on virtualization , go with hypervisor . if your focus is storage, go with something like truenas. Can Truenas do light virtualization in a pitch - yes, sure. is it same feature equivalent as say proxmox, not in my experience.

I use both. they are differently focused tools.
That's a good idea, I haven't looked at Proxmox but reading about it now I see that ZFS on Linux is included in the kernel so it looks like a good option. I like the web UI which is something I've been missing from SmartOS (it's possible with something like FiFo but for a home use case with a single server it's a bit over the top I think).

I'd like to have a hypervisor for my different servers so I think I might give Proxmox a try, I don't think TrueNAS is right for me. Thanks for your input!
 
Last edited:

EngChiSTH

Member
Jun 27, 2018
95
39
18
Chicago
That's a good idea, I haven't looked at Proxmox but reading about it now I see that ZFS on Linux is included in the kernel so it looks like a good option. I like the web UI which is something I've been missing from SmartOS (it's possible with something like FiFo but for a home use case with a single server it's a bit over the top I think).

I'd like to have a supervisor for my different servers so I think I might give Proxmox a try, I don't think TrueNAS is right for me. Thanks for your input!

Sounds good, Proxmox has a healthy community and pretty good support. Everything I do beyond initial install is through web UI (node/cluster management, VM managed, console, storage management, failovers, etc). May be one item I had to do in command line when I did the storage format conversion of the disks but even that I think is now in the UI within migrate disk command that also allow you to switch formats. Very good software.

TrueNAS is something different - to me it is about configuring volumes, datasets, caching, encryption on your data, degree of redundancy ,etc - not a virtualization platform. many people run TrueNas from under Proxmox , I dont as I like to separate storage function (trueNAS and QNAP I have) from compute functions (proxmox cluster in the homelab)
 

sko

Member
Jun 11, 2021
52
27
18
You might want to take a look at Danube Cloud (Danube Cloud Community Edition), which is in essence smartOS bundled with pre-configured zones for a management GUI, zabbix and basic services like DNS/DHCP.

We are using smartOS and vanilla FreeBSD here and running zones/jails and bhyve on them. There are a few KVM VMs on one smartOS host left, but with bhyve being _much_ more lightweight, modern and directly providing a console (or vnc via the UEFI firmware), all new HW-VMs over the last few years were installed on bhyve...

I have Danube Cloud running on a dev/test Server, but if I wouldn't have already built an infrastructure around our smartOS and FreeBSD ecosystem as well as zabbix monitoring, Danube would definitely be high on my list. It basically resembles parts of what I've built here manually, although as with any GUI you are much more limited and restricted and need to adapt your workflows/infrastructure to the capabilities of the GUI.

As for TNC: I also support 2 TNC systems in the home office of my boss (together with an OPNSense gateway). Yes, they are all rock-solid and work like a charm and it might be nice to have a GUI with pretty graphs and all, but it also very much restricts what you can do with the system. Especially because there is a lot of weird, non-standard stuff going on underneath that often doesn't comply with the way FreeBSD is doing things normally. So despite it having FreeBSD underneath, you often have to build around all that automation for the GUI to achieve relatively simple goals.
E.g. setting up a dead-simple nginx reverse proxy /w SSL on OPNSense was catastrophic - it took WAY more time than just doing it manually and broke at every possible occasion. I replaced it with a simple nginx + acme.sh jail which was completely set up in less than 10 minutes and 'just works'™.

So if you want/need the flexibility of a true multi-purpose system, go with vanilla FreeBSD or smartOS. If you want/need a shiny GUI and don't care about the many restrictions, go with TrueNAS and/or Danube Cloud

Regarding ZFS: Between FreeBSD and smartOS/illumos you can freely import/export and send/receive any pool or dataset between those OSes. You might get warnings about unsupported features or old pool version, but as long as you don't rely on any of those newly added features, you are fine.
I've been moving pools and datasets between FreeBSD and illumos for several years now and never had any issues. (I hope it stays this way with FreeBSD moving to ZoL codebase...)

SMB on smartOS/illumos is very straightforward as @gea already pointed out, thanks to the native CIFS-server and "sharesmb" ZFS-property. On FreeBSD you need a minimal samba configuration for it to work and of course you have to deal with the usual oddities of samba, (e.g. its user management).
 
  • Like
Reactions: gb00s

knorrhane

New Member
Dec 17, 2016
8
0
1
42
You might want to take a look at Danube Cloud (Danube Cloud Community Edition), which is in essence smartOS bundled with pre-configured zones for a management GUI, zabbix and basic services like DNS/DHCP.
I have never heard about Danube but definitely looks interesting, it looks like the GUI has most of the things I'm looking for including setting up a bhyve VM which is nice. Thanks for the tip!