Cockpit ZFS Manager [0.1.1.158 Now Available]

Discussion in 'Linux Admins, Storage and Virtualization' started by optimans, Aug 31, 2019.

  1. optimans

    optimans Member

    Joined:
    Feb 20, 2015
    Messages:
    33
    Likes Received:
    26
    Cockpit ZFS Manager
    An interactive ZFS on Linux admin package for Cockpit.

    WARNING!

    Cockpit ZFS Manager is currently pre-release software.

    Use at own risk!

    Not recommended for use on production systems.
    Ensure all critical data is adequately backed up before use.


    Requirements
    • Cockpit: 201+
    • Samba: 4+
    • ZFS: 0.8+
    Tested Distributions:
    • CentOS Linux 8.0
    • Debian 10.0
    • Oracle Linux Server 8.0
    • Proxmox VE 6.0
    • Red Hat Enterprise Linux 8.0
    • Ubuntu 18.04 LTS
    • Ubuntu 19.10

    0.1.1.158 Now Available:

    github.com/optimans/cockpit-zfs-manager




    Screenshots (Out of date):

    01.OPTVMFS-01.StoragePools.png 02.OPTVMFS-02.StoragePools.png 03.OPTVMFS-01.FileSystems.png 04.OPTVMFS-01.Snapshots.png 05.OPTVMFS-01.Status.png 06.OPTVMFS-01.StoragePoolDestroy.png 07.OPTVMFS-01.StoragePoolCreate.png 08.OPTVMFS-01.StoragePoolCreateNoWWN.png 09.OPTVMFS-01.StoragePoolImport.png 10.OPTVMFS-01.StoragePoolImport2.png 11.OPTVMFS-01.StoragePoolActionsMenu.png 12.OPTVMFS-01.StoragePoolConfigure.png 13.OPTVMFS-01.StoragePoolConfigureMultiHost.png 14.OPTVMFS-01.FileSystemCreate.png 15.OPTVMFS-01.FileSystemCreateEncrypted.png 16.OPTVMFS-01.FileSystemActionsMenuRoot.png 17.OPTVMFS-01.FileSystemConfigureRoot.png 18.OPTVMFS-01.FileSystemConfigureChild.png 19.OPTVMFS-01.FileSystemUnmountRoot.png 20.OPTVMFS-01.FileSystemSambaEnableRoot.png 21.OPTVMFS-01.FileSystemRename.png 22.OPTVMFS-01.FileSystemEncryptedStatus.png 23.OPTVMFS-01.FileSystemEncryptedActionsMenu.png 24.OPTVMFS-01.FileSystemEncryptedUnlock.png 25.OPTVMFS-01.SnapshotCreate.png 26.OPTVMFS-01.SnapshotCreateAutoGenerateDate.png 27.OPTVMFS-01.SnapshotsActionsMenu.png 28.OPTVMFS-01.SnapshotsClone.png 29.OPTVMFS-01.FileSystemCloneActionsMenu.png 30.OPTVMFS-01.StatusPoolActionsMenu.png

    More screenshots: Post #3
     
    #1
    Last edited: Dec 7, 2019 at 12:36 AM
    EluRex, sfu420, microserf and 3 others like this.
  2. dragonme

    dragonme Active Member

    Joined:
    Apr 12, 2016
    Messages:
    282
    Likes Received:
    28
    been running zfs on OS X since 2008 ish.. while I would not consider myself a massive power user, I would be willing to beta test a bit

    I was a software engineer / quality assurance and IV&V director of test in a previous life

    not familiar with the cockpit project.. link?

    currently my OS X zfs server is somewhat sidelined and a backup of sorts

    currently running a ESXI all in one

    napp-it storage both direct to the network for NAS and to esxi for nfs backing

    have several linux VMs running various tasks..
     
    #2
  3. optimans

    optimans Member

    Joined:
    Feb 20, 2015
    Messages:
    33
    Likes Received:
    26
    Here are some more screenshots

    31.OPTVMFS-02.StatusPoolVDEVAdd.png 32.OPTVMFS-02.StatusPoolVDEVAdd2.png 33.OPTVMFS-01.StatusPoolTRIM.png 34.OPTVMFS-01.StatusVDEVActionsMenu.png 35.OPTVMFS-01.StatusDiskActionsMenu.png 36.OPTVMFS-02.StatusDiskAttac6.png 37.OPTVMFS-02.StatusDiskAttach2.png 38.OPTVMFS-02.Status.png 39.OPTVMFS-02.StatusActionsMenu.png 40.OPTVMFS-02.StatusPoolDegradedDiskOffline.png
     
    #3
  4. optimans

    optimans Member

    Joined:
    Feb 20, 2015
    Messages:
    33
    Likes Received:
    26
    @dragonme Thanks for response, I have added the link as requested and some more information/screenshots.
     
    #4
  5. dragonme

    dragonme Active Member

    Joined:
    Apr 12, 2016
    Messages:
    282
    Likes Received:
    28
    sounds like a great project.. and very timely now that joyent have abandoned solars as their head for zfs development and moved linux to the head spot moving forward as the lead operating system

    my thoughts on zfs

    thus far I have dabbled with many front ends, Sun's zfs storage server (one of the better ones), freenas, napp-it etc

    one glaring issue with them all is that administration of ZFS from a GUI only interface at some point almost always leads an admin to do something stupid...

    1> because its not always evident WHAT or HOW something is actually going to be done.. so graphically it all looks good but the action fails to do what the admin THOUGHT was going to happen.. and MANY things in ZFS are un-reversable are require the full destruction and rebuild at best or DATA LOSS at the other end

    2> GUI administration tends to lower the bar and stagnates or eliminates the learning curve that I feel anyone managing a ZFS array needs to learn

    SOLUTIONS

    my suggestions then for any GUI front end to ZFS is that

    1> any ACTION features of the GUI should show the actual terminal commands that the pending operation will accomplish and be editable so the admin can tweek it if necessary.

    2> clearly warn when a pending activity will result in a irreversible data structure change or result in data destruction. and show a pre and post view of what is pending before execute is pressed.

    3> monitoring .. I think that none of the current front ends really power use dtrace. sure they provide pretty graphs (some of them) but its usually rather simplistic raw data.. not really curated into a collection of graphs of analyzed raw data to help solve an actual bottleneck or other issue. Suns storage servers again are better here and it stands to reason .. they have been at the ZFS and dtrace game since the beginning since they invented both. and their ZFS skunkworks had some really great dtrace graphs and scripts.. I watch as they guy started yelling at a 36 drive shelf and the graphing dtrace screen was showing latency jitter on the drives in real time showing that the noise of his yelling was effecting the array .. powerful stuff

    Lastly... on linux

    I hate doing basic house keeping and setup of linux from the command line (my spoiled OS X upbringing) so a great suite of admin tools to properly configure and monitor the linux host (like webmin but better) would be a welcome breath of fresh air.
    tuning of the underling operating system network for example is just as important as turning the knobs on tuning ZFS in order to fully realize high performance operation ...

    I assume this runs on server linux without a gui front end but generate a web gui for Cockpit...

    I look forward to giving it a spin...

    bare metal and esxi ova .. can test both... and I can put it on some older commodity hardware (like most home users)

    this will be a winner if its feature complete, fully free, and not cut down like other solutions and of course being the head for openzfs .. should remain as cross platform pool capable as possible... or again, warn a user before making or changing a pool feature flag that will limit it portability .. this is one of the most important benefits of ZFS.. being able to yank 36 drives out of one shelf on zfs and move it over to a completely different installation running a different operating system and import it like it was native...
     
    #5
  6. dragonme

    dragonme Active Member

    Joined:
    Apr 12, 2016
    Messages:
    282
    Likes Received:
    28
    That aforementioned Linux migration added Delphix's impressive array of OpenZFS developers to the large contingent already working on ZFS on Linux. In November, the FreeBSD project announced its acknowledgment of the new de facto primacy of Linux as the flagship development platform for OpenZFS. FreeBSD did so by rebasing its own OpenZFS codebase on ZFS on Linux rather than Illumos. In even better news for BSD fans, the porting efforts necessary will be adopted into the main codebase of ZFS on Linux itself, with PRs being merged from FreeBSD's new ZoL fork as work progresses.

    sorry .. old age is a bitch.. I meant Delphix.. joyent probably too as openzfs is pretty much lead by Delphix
     
    #6
  7. dragonme

    dragonme Active Member

    Joined:
    Apr 12, 2016
    Messages:
    282
    Likes Received:
    28
    gea had a good post about it.. and rather than link it here it is..

    but I have read in multiple places that Zol has now become the head for openZFS development

    a final brake if you will from solaris and sun... its been a long time coming

    when they dropped community support from opensolaris and denied access to zfs codebase and it was forked (for legal reasons) into the non compatible with solars openzfs.. the writing was on the wall... with Linux proliferating into the server space its only natural that it will begin to dominate the storage space as well

    here are gas's comments


    It's all about new ZFS features.
    In the last years they were mainly developed from Illumos related firms like Delphix, Joyent or Nexenta where most ZFS developers landed when they left Oracle after overtaking Sun.

    The common Open-ZFS platform ensures that new features could be part of ZFS in Free-BSD or ZoL

    Now the situation is different.
    The Linux platform is much much larger than the Unix variants Free-BSD or Solarish. ZFS is now also the leading filesystem on Linux with many firms developing around ZFS.

    A typical example is ZFS encryption. While it is based on the OpenSolaris/Illumos bits where it was nearly ready (Oracle published it with Solaris 11) and not Linux to avoid GPL problems, it is finalized by Datto on ZoL and is now in the process of beeing ported to Illumos and Free-BSD.

    Porting it to Illumos and then from Illumos to Free-BSD makes no sense. Porting form the OS of origin to a target OS is always needed and should be done directly.

    OS of origin of new ZFS features is more and more ZoL and to a lesser extend Illumos.
    After all this is good news for ZFS on all platforms and this means today Free-BSD, Illumos, OSX, propably Windows and Linux.

    Up to now new ZFS features were mostly available first on Illumos and with a delay on others. Whats happening is that more and more are first published on ZoL and with a delay on the others.
     
    #7
  8. optimans

    optimans Member

    Joined:
    Feb 20, 2015
    Messages:
    33
    Likes Received:
    26
    Yes, Cockpit has its own web service, can run headless; as the project continues, they are adding more and more controls to it, can even use Cockpit as a virtualization host to manage your VMs.

    This is exactly what I'm looking for. I've been testing and trashing real and virtual servers/disks on my end, but need different configurations to prove it works as should.

    May be able to add an expert/advanced mode that user can select so they can see command info if need be.

    Some of the modals do include warning text, however could probably add a switch toggle to critical actions (VDEV add/remove, disk offline/replace/labelclear, upgrade, etc) so they can't continue without acknowledging.

    Ease of use as and cross compatibility is definitely part of the plan.

    I think this is great news, it shows that ZFS is very much alive, and the power of the community as a whole.
     
    #8
  9. dragonme

    dragonme Active Member

    Joined:
    Apr 12, 2016
    Messages:
    282
    Likes Received:
    28
    I think ZFS still has a bright future...

    most important of a file system is maturity and ZFS has more than a decade under its belt

    I have been running ZFS on multiple platforms using cheap consumer grade drives in against standards single disk 8 drive raidz volumes and even 6-8 wide volumes running NO redundancy.. have not lost a single file in over 10 years

    only question mark moving forward as openzfs has had to refactor and reverse engineer most of suns code... will the forward development be as thought out, stable, and capable... so far it looks good
     
    #9
  10. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,261
    Likes Received:
    750
    The case is diferent.

    Up from ZFS v28/5 (OpenSource OpenSolaris) Oracle developped its own ZFS line Solaris 11,x. While native Oracle ZFS is still the fastest and most feature rich ZFS , Open-ZFS is improving its own fork of (Opensource) ZFS. No (illegal?) reverse engineering but independent software development.
     
    #10
  11. Elliott

    Elliott New Member

    Joined:
    Sep 3, 2019
    Messages:
    2
    Likes Received:
    0
    #11
  12. Raice

    Raice Member

    Joined:
    Jul 12, 2017
    Messages:
    40
    Likes Received:
    5
    Wow! Great project! How can I try it?
     
    #12
  13. Robertejes

    Robertejes New Member

    Joined:
    Sep 9, 2019
    Messages:
    2
    Likes Received:
    0
    I'd like to try testing too. Have some QA experience with oVirt
    Just hiding it from everybody. I also have a good experience with testing of paraphrasing tool pages(for example, Paraphrasing Tool Online for Free). If you look at it, it works as business owner needs.
     
    #13
    Last edited: Sep 12, 2019
  14. optimans

    optimans Member

    Joined:
    Feb 20, 2015
    Messages:
    33
    Likes Received:
    26
    Currently doing code clean up at the moment :eek:. Still hoping to release a beta in about a weeks time. Will update details on first post when available.
     
    #14
    Last edited: Oct 1, 2019
  15. optimans

    optimans Member

    Joined:
    Feb 20, 2015
    Messages:
    33
    Likes Received:
    26
    Not using JSON yet, but will have a look at it, would probably make zpool status coding easier.

    I had a look at the GitHub issue a little while ago; looks like its labelled as blocked, wonder if there will be any review in near future.
     
    #15
  16. Wolfstar

    Wolfstar Member

    Joined:
    Nov 28, 2015
    Messages:
    60
    Likes Received:
    28
    Tossing a "pick me!" on here to track it. Just migrated to Ubuntu 18.04 off of OMV because if I'm going to manage ZFS on the command line anyhow, might as well do it on something with more options in general. I know that rules me out from a version perspective, but I don't mind updating to 19.04 or (when it comes out) 19.10 interim for testing.
     
    #16
  17. optimans

    optimans Member

    Joined:
    Feb 20, 2015
    Messages:
    33
    Likes Received:
    26
    Thought I'd better give a quick update:

    Nearly finished with code work; bug fixes and trying to maximize compatibility. NFS UI support has been added plus more. Almost ready for final testing. Shouldn't be too much longer.
     
    #17
    manxam and Wolfstar like this.
  18. alazare619

    alazare619 New Member

    Joined:
    Oct 5, 2019
    Messages:
    1
    Likes Received:
    0
    Do you have a github or gitlab...with this + cockpit in general thats everything thats needed for a roll your own very effecient nas..super excited i'd love to give it a try and submit any potential PR's against as needed
     
    #18
  19. optimans

    optimans Member

    Joined:
    Feb 20, 2015
    Messages:
    33
    Likes Received:
    26
    Yes, have got a Github account, and will be publishing to it when ready. Will admit Github is new area for me so will need to get my head around it.

    When I started making this, it was purely for personal use (trying to get away from Solaris), but now with the apparent interest generated within this forum, it looks like it could be the start of something greater than that I first thought.
     
    #19
    flyemsafe, Patriot and MaxCFM like this.
  20. Fukuzawa Katsuo

    Fukuzawa Katsuo New Member

    Joined:
    Oct 11, 2019
    Messages:
    1
    Likes Received:
    0
    big fan for zfs!
    I am a practitioner of geographic information science, and have used zfs as a storage system to complete graphics rendering for a couple of years.
    using CLI to manage pool or filesystem, Which is of course quite cumbersome.
    glad to help you test, hope to see your great work.
     
    #20
Similar Threads: Cockpit Manager
Forum Title Date
Linux Admins, Storage and Virtualization MegaRAID Storage Manager in x64 DEB distros Oct 1, 2012

Share This Page