Poolsman - ZFS Web GUI for Linux based on Cockpit

[PoolsmanTeam]

Hello,

we are working on new ZFS GUI called Poolsman (www.poolsman.com) and would like to invite you to join our Early Access Program (EAP). Poolsman is intended to give an easy, graphical way for managing ZFS on commonly used Linux systems (including Debian, Ubuntu, RHEL, Rocky, OpenSUSE, Arch, Proxmox and other distros). It's implemented as a Cockpit module and provides familiar user experience (if you worked with this admin panel before).

The first Poolsman preview release is expected on September 30, 2022 and will include:

1. Support for main ZFS features (Pools/Datasets/Snapshots/Clones management, including Compression/Deduplication/Encryption);
2. Disks monitoring (temperature, health, S.M.A.R.T.);
3. Dashboard with the top level information about ZFS RAM usage, disks and pools;
4. Unique Command Preview feature (ability to review each CLI command before execution);
5. Light and dark UI theme.

We are thinking of adding more features later, including data replication and scheduled tasks (trim, scrub and S.M.A.R.T. checks), as well as developing additional modules for Samba, NFS, iSCSI shares and Cloud Sync Tasks (Backblaze, S3, etc).

During EAP stage we are going to provide Poolsman by various plans depending on support level. There are no free version yet, but we tried to set affordable prices. It's because we are independent, self-funded team and need a way to support our development. We are also not ready to disclose source codes right now. Both of these things can be changed later with the growth of this project and your interest.

Would be very happy to get any feedback and hear about your use cases, where you are missing a good ZFS management GUI.

Here's the link to the project's website where you can get more information (and screenshots):

www.poolsman.com

 

[PoolsmanTeam]

Hi all,

We are pleased to announce the first Poolsman preview release. It already provides a lot of features, including:

1. Pools management (create/destroy, currently pool creation is limited to 1 VDev);
2. Datasets (both Filesystems and ZVols), Snapshots and Clones management (create/destroy with full support of compression and deduplication);
3. Disks monitoring (temperature, health, S.M.A.R.T.);
4. Dashboard with the top level information about ZFS RAM usage, disks and pools;
5. Unique Command Preview feature (ability to review each CLI command before execution);
6. Light and dark UI theme.

Probably the best way to meet with all features is to explore our Screenshots Gallery.

With preview 2 release, that will become in the end of October, we are going to eliminate existing limitations and add more features, including:

1. Pool expansion and Disks replacement.
2. Ability to add more than 1 VDev during Pool creation.
3. Editing parameters of existing datasets.
4. Encryption configuration support (passwords, hex/bin file keys, lock/unlock).

 

[vudu]

Nice work. You may consider adding ssd wear rate to the topology page.

 

[PoolsmanTeam]

Nice work. You may consider adding ssd wear rate to the topology page.

Thank you. Yeah, we will add SSD wear rate later. The problem there is that SSD manufactures use different S.M.A.R.T. attributes for wearout (needs some time for research).

 

[PoolsmanTeam]

Hi everyone,

just want to inform that next release with new features is slightly delayed and will become available in November. The priority now is working on compatibility issues of EAP users. We've also received multiple questions regarding Trial and Lifetime License via contact form. We are going to provide a way for getting a Trial license with the next release and will prepare various Lifetime licenses (depending on functions) with the final 1.0 release. We will also propose a solution for converting EAP licenses to Lifetime licenses. Stay tuned!

 

[PoolsmanTeam]

Hello everybody,

we are pleased to present the 2nd Poolsman preview release with various new features and significant bug fixing based on first user experience. It includes:

1. Editing settings of existing datasets (both file systems and volumes).
2. Renaming of existing snapshots.
3. Ability to safely disconnect pool from the system (pool export).
4. Theme settings are persistent now and stored in a browser local storage.
5. Data validation in all forms has been significantly improved.
6. Fixed issues with various antivirus and firewall systems, that prevended loading Blazor WebAssembly files in a web browser.

As before, every action in UI generates a shell command that can be reviewed before execution.

Today we are also starting to provide a free 7-day trial access upon request in manual mode (it will be automated later). If you'd like to get a trial now, please register on Poolsman website and ask for a trial key via Contact form.

The next preview release is expected in the end of December. It's going to bring huge improvements in pools and disks management (pools expansion, disks replacement). Keep watching!

 

[PoolsmanTeam]

Hello,

today's Preview 3 release is not big, but very helpful in some cases. We are still working on a massive set of features in part of pools expansion and disks replacement, and decided to leave it for the next release. The current Preview 3 version includes:

1. Checking that ZFS is installed. If it's not installed, Poolsman proposes to open the installation manual for your operating system.
2. Checking that ZFS is running. If not, Poolsman proposes to start it and allows to preview the start command.
3. Fix of various issues on RHEL 8 based distros (including Rocky Linux 8) related with an old version of lsblk tool.
4. Fix of various issues related with the case when the system language is not English.

Wishing you a Happy New Year from the Poolsman Team. Thanks to all our users, it's a pleasure to make Poolsman for you.

 

[cojones]

Bought an EAP license recently and so far it's great. Have you any plans to integrate with an automatic snapshotting / zfs send solution like sanoid / syncoid?

 

[PoolsmanTeam]

Bought an EAP license recently and so far it's great. Have you any plans to integrate with an automatic snapshotting / zfs send solution like sanoid / syncoid?

Thank you for your feedback. Yes, we do have plans to implement automatic snapshots and scheduled replication. Sanoid/Syncoid probably will be the first supported tool for that.

 

[cojones]

Thank you for your feedback. Yes, we do have plans to implement automatic snapshots and scheduled replication. Sanoid/Syncoid probably will be the first supported tool for that.

Do you have a roadmap somewhere to see what's coming up?

 

[PoolsmanTeam]

Do you have a roadmap somewhere to see what's coming up?

We don't have detailed roadmap yet. Currently we are providing monthly EAP preview releases with new features, that we managed to do during this time. We expect to release the 1st final version this year (probably in the second half of the year).

There are main missing features, that we are going to add during EAP period:

1. Pool expansion and disks replacement.
2. Pool creation wizard improvements (ability to add more than 1 VDev, advanced pool settings).
3. Encryption configuration support (passwords, hex/bin file keys, lock/unlock).
4. Pool maintenance (trim, scrub management).
5. Advanced settings for datasets (file systems and zvolumes).
6. UI polishing and rethinking the structure of some pages in order to achieve better user experience.
7. DEB/RPM/AUR packages and repositories.

Automatic snapshots and replication functionality most likely will be implemented after the 1st final release, but it's already in our plans and expected this year too. It will be the next biggest improvement. We'd like to add more features and modules after that (Samba/NFS/iSCSI, e-mail notifications, etc.) and hope that we will get enough users interested in it.

If you miss some features right now, please tell us about that and we will think about adding that into our plans and/or changing the priorities.

 

[cojones]

This is great stuff, especially if integrated with automated snapshotting and replication. It makes using cockpit for ZFS NAS boxes far more viable.

 

[Beardmann]

Hi there, this looks promising, so I bought an EA license, and added it to my Cockpit, and I have a few issue, mainly when trying to list disks or one of my pools which happen to be a "draid3:19d:24c:2s-0" with a mirrored special device and two disks for cache...
I am willing to provide logs etc. in a hope that the project can include this "special" type of zpool which I found to be best suited to my workloads..
But is this the right place to post such issues?

 

[PoolsmanTeam]

Hi there, this looks promising, so I bought an EA license, and added it to my Cockpit, and I have a few issue, mainly when trying to list disks or one of my pools which happen to be a "draid3:19d:24c:2s-0" with a mirrored special device and two disks for cache...
I am willing to provide logs etc. in a hope that the project can include this "special" type of zpool which I found to be best suited to my workloads..
But is this the right place to post such issues?

Hi Beardmann, we are glad to see you among Poolsman users. Yes, it's the right place to post such issues. The reason is that DRAID configurations are not supported. We've prepared a hotfix that allows to display DRAID configurations. Please try to download and install the new release (v.0.3.1.0), that includes this hotfix.

 

[Beardmann]

Hi Beardmann, we are glad to see you among Poolsman users. Yes, it's the right place to post such issues. The reason is that DRAID configurations are not supported. We've prepared a hotfix that allows to display DRAID configurations. Please try to download and install the new release (v.0.3.1.0), that includes this hotfix.

Great with such fast response
I downloaded version 0.3.1.0 and unpacked it over the existing one, but sadly it didn't fix my issues with my draid pool, and I can see that I actually also have an issue showing the one of my other pools which is just a simple "raid0" pool with two devices...
As soon as I try to show the topology or disks from pools aggr0 or vms it kicks out the error at the bottom...
Also in the main dashboard it shows all my pools, but the "Disks" view just cycles and I am again presented with the error at the bottom...

Just as a reference, there are the pools from the server in question:

root@box:/usr/share/cockpit# zpool status
  pool: aggr0
state: ONLINE
  scan: scrub repaired 0B in 1 days 09:48:51 with 0 errors on Mon Jan  9 10:12:53 2023
config:

    NAME                   STATE     READ WRITE CKSUM
    aggr0                  ONLINE       0     0     0
      draid3:19d:24c:2s-0  ONLINE       0     0     0
        0a-0               ONLINE       0     0     0
        0a-1               ONLINE       0     0     0
        0a-2               ONLINE       0     0     0
        0a-3               ONLINE       0     0     0
        0a-4               ONLINE       0     0     0
        0a-5               ONLINE       0     0     0
        0a-6               ONLINE       0     0     0
        0a-7               ONLINE       0     0     0
        0a-8               ONLINE       0     0     0
        0a-9               ONLINE       0     0     0
        0a-10              ONLINE       0     0     0
        0a-11              ONLINE       0     0     0
        0a-12              ONLINE       0     0     0
        0a-13              ONLINE       0     0     0
        0a-14              ONLINE       0     0     0
        0a-15              ONLINE       0     0     0
        0a-16              ONLINE       0     0     0
        0a-17              ONLINE       0     0     0
        0a-18              ONLINE       0     0     0
        0a-19              ONLINE       0     0     0
        0a-20              ONLINE       0     0     0
        0a-21              ONLINE       0     0     0
        0a-22              ONLINE       0     0     0
        0a-23              ONLINE       0     0     0
    special
      mirror-1             ONLINE       0     0     0
        nvme01-part1       ONLINE       0     0     0
        nvme02-part1       ONLINE       0     0     0
    cache
      sdc1                 ONLINE       0     0     0
      sdd1                 ONLINE       0     0     0
    spares
      draid3-0-0           AVAIL
      draid3-0-1           AVAIL

errors: No known data errors

  pool: bpool
state: ONLINE
  scan: scrub repaired 0B in 00:00:01 with 0 errors on Sun Jan  8 00:24:04 2023
config:

    NAME                                      STATE     READ WRITE CKSUM
    bpool                                     ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        a45db0bd-8c51-7445-8626-5e9a06d170c6  ONLINE       0     0     0
        f3241a1e-a49c-4434-ad1a-84e8211f9cac  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
state: ONLINE
  scan: scrub repaired 0B in 00:00:30 with 0 errors on Sun Jan  8 00:24:33 2023
config:

    NAME                                      STATE     READ WRITE CKSUM
    rpool                                     ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        5eb513ba-4e98-e44a-93f9-5506135bc2b9  ONLINE       0     0     0
        5d5b0c5d-7c56-4055-958b-b03fa27691ab  ONLINE       0     0     0

errors: No known data errors

  pool: vms
state: ONLINE
  scan: scrub repaired 0B in 00:09:07 with 0 errors on Sun Jan  8 00:33:11 2023
config:

    NAME            STATE     READ WRITE CKSUM
    vms             ONLINE       0     0     0
      nvme01-part2  ONLINE       0     0     0
      nvme02-part2  ONLINE       0     0     0

errors: No known data errors

root@box:/usr/share/cockpit# apt list cockpit-system
Listing... Done
cockpit-system/jammy-backports,jammy-backports 283-1~bpo22.04.1 all

root@box:/usr/share/cockpit# zfs -V
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.4-0ubuntu0.1

root@box:/usr/share/cockpit# uname -a
Linux box 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

One last thing... I also use the /etc/zfs/vdev_id.conf in order to create the device-names in the aggr0 pool... they match the slots in the shelf this way...

Hope this helps? I'm not sure where the logs are stored, but let me know if you need me to pull something...

/B

 

[PoolsmanTeam]

One last thing... I also use the /etc/zfs/vdev_id.conf in order to create the device-names in the aggr0 pool... they match the slots in the shelf this way...

Got it. It might be the root cause Such device-names configuration hasn't been tested before. We'll try to reproduce and test it. If any, will ask additional logs from you be email. Hope to come back with a reply in few days.

 

[Beardmann]

Got it. It might be the root cause Such device-names configuration hasn't been tested before. We'll try to reproduce and test it. If any, will ask additional logs from you be email. Hope to come back with a reply in few days.

Yep that must be it, because the vms pool has the same issue.. but the rpool/bpool works as expected...

/B

 

[tssrshot]

So this might be more in the line of “best practice,” but I figured you built it…you already know way more than me.

I’m using the early adopter, as of today. Loads fine, works well.

I have a beast server running Ubuntu 22.04, with 20 x SSD 1TB Drives and 4 x NVME 1TB Drives.

I can make a pool with the 20 drives in Z1. No problem. I can make another pool with the NVME.

Is there something I should be doing to use the NVME for “the greater good” with the Main VDEV? I didnt have any options to select during the initial pool build for ARC, Special, etc. Just trying to maximize IOPS for a DB workload In our dev environment.

Any ideas? Right now, I can destroy, and rebuild at will, but they are on my back to hurry up too, so they can get to their DevOps work.

 

[PoolsmanTeam]

Yep that must be it, because the vms pool has the same issue.. but the rpool/bpool works as expected...

/B

Hi Beardmann, probably we've resolved all issues with your deployment in private emails. Please let us know if you have anything else.

 

[PoolsmanTeam]

I have a beast server running Ubuntu 22.04, with 20 x SSD 1TB Drives and 4 x NVME 1TB Drives.

Oh, you have a great machine that should work well with DB workload, but of course it depends on your expectations.

I can make a pool with the 20 drives in Z1. No problem. I can make another pool with the NVME.

The general recommendation from us is to not to use more than 10 drives in a disk group (VDEV). For your amount of SSD drives we would suggest:

1. 10xMirrors (if you'd like to maximize iops when the cache is not used)
2. 5xRaidZ2 groups of 4 drives (more redundancy but twice less iops than mirrors configuration, when the cache is not used)
3. 3xRaidZ2 groups of 6 drives and 2xSpare (more effective space, but less iops than 4 drives groups)
4. 2xRaidZ2 groups of 10 drives (much more effective space, but even less iops)

The RaidZ2 groups (VDEVs) of 4, 6 or 10 drives are the most optimal, because the number of data drives in these groups has power of two (2, 4 and 8 respectively). It allows ZFS to effectively allocate logical blocks (which sizes has power of two) across multiple data drives.

Is there something I should be doing to use the NVME for “the greater good” with the Main VDEV?

Probably it's a good idea to use them as SLOG devices for a database workload. Assuming that you can't allow any lost of your data, you should go with 2xMirrors there. But your NVME drives must have PLP (power loss protection) in order to be used as SLOG devices. Without PLP you might get the performance comparable even with HDD drives, because DRAM cache on your drives will not be used without PLP. Please check that your NVME drives have PLP support.

I didnt have any options to select during the initial pool build for ARC, Special, etc.

We are working on UI that would allow to create pools that includes multiple disk groups (VDEVs). It's slightly delayed, but should be released in February. You might also consider buying a license with support tickets in order to get more consulting about your deployment (and we could also help to create pool configurations that are not supported yet through UI).