ZFS REST API interest

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
Hi,

I like to build stuff - and I have built a .NET Core ZFS library.

This library I have used in a .NET Core ZFS REST API service, so it provides a restful interface for manipulating zfs - which would allow someone to build a UI to control ZFS on a given server.

The API does not have to run on the server that has the ZFS pool by itself, but then it will use ssh and passwordless sudo to invoke commands - it runs perfectly both on Windows and Linux.

If its running on the server where the ZFS lives, then it uses linux PAM to authenticate the user, but still uses the service account to interface with ZFS.

The plan is to pass on the credentials used and invoke the ZFS command in the context of the user, but I just need to figure out how to do that.

The API is by all means not fully complete, but right now I have the following implemented:
1664015873395.png

An example on how to use the API from the command like would be:
Bash:
curl -X 'GET' \
  'http://zfsdev.root.dom:5000/api/zfs/datasets/tank%2Fmyds/properties/atime' \
  -H 'accept: text/plain'
Which would return the following:
JavaScript:
{
  "data": {
    "name": "atime",
    "value": "on",
    "source": "local"
  },
  "status": 0,
  "errorText": null
}
Or if you wanted to set a single property:
Bash:
curl -X 'POST' \
  'http://zfsdev.root.dom:5000/api/zfs/datasets/tank%2Fmyds/properties' \
  -H 'accept: text/plain' \
  -H 'Content-Type: application/json' \
  -d '[
  {
    "name": "atime",
    "value": "off"
  }
]'
This post is merely to gauge interest in an API like this and if someone would like to build a UI based on this - obviously a web ui could use the rest api directly, but it is also possible to build a desktop UI using the ZFS library I built directly (.NET Core)

Please PM me if you are interested.

P.S. Right now all the source code is hosted on my own private git repository, so I have no easy way of sharing the source code, but if enough interest exist, then I am happy to make a public github repository with the source code - although I will probably have to clean it up a bit, so I will not be too embarrased to show the code :-D

P.P.S - I can probably open up for testing the API directly on my own server, if you just want to play around with the API - right now it uses basic authentication, since its simple.

API code is on github
ZFS Library code is also on github
 
Last edited:

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
Freenas/truenas and napp-it hate this trick ;)
I don't have anything against any of those - I just always wanted something for any platform - and I think this API would run fine on freebsd/linux/solaris etc.

Sometimes you just want to install a plain linux - but still have a UI to control your ZFS - without the fullblown platform that Truenas etc gives you.
 
  • Like
Reactions: Aluminat

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
Anyone has any ideas on where I could ask for interest as well?
Its not like I am being flooded with PM's here :)
 

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
Why not ask the zol project directly?
I did not consider this, because its not direcly related to ZoL - and more of a "who uses zfs and would like an API to manage it" - but I might give it a try - so thanks for the suggestion.
 

uldise

Active Member
Jul 2, 2020
173
53
28
are you looked at unraid community? there is ZFS plugins for unraid, so i think there should be something like API..
 

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
I have never used unraid - and I doubt the unraid zfs plugin is usable outside of unraid.
Also I an not looking for an API - I have built an API :)
 

gea

Well-Known Member
Dec 31, 2010
2,831
984
113
DE
The common interface to zfs on all platforms are the zpool and zfs commands.
Any API must double and follow their functionality and must care about the ongoing development.

I always use zfs and zpool directly (in napp-it).
Any new zfs option and bugfix is available without delay in these commands.
 

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
The common interface to zfs on all platforms are the zpool and zfs commands.
Any API must double and follow their functionality and must care about the ongoing development.

I always use zfs and zpool directly (in napp-it).
Any new zfs option and bugfix is available without delay in these commands.
Hi,

I can understand why you use zfs directly in napp-it, but lets say someone wanted to build another napp-it - or perhaps not as advanced - just a simpleish GUI to manage a linux server - if that is the case and you do not need _ALL_ the functionality of zpool/zfs commands, then an API might make sense - also considering having a more "advanced" entry point to zfs/zpool enabled you to do things that cannot directly be done with zfs/zpool directly without having to resort to scripting - an example could be wildcard deletion of snapshots - that is something that I have wanted for a very long time. In the API i have built you just do a :

Bash:
curl -X 'DELETE' \
  'http://zfsdev.root.dom:5000/api/zfs/datasets/tank%2Fmyds%2FROOT/snapshots/202207?isExactName=false' \
  -H 'accept: text/plain'
Or delete a dataset recursively

Bash:
curl -X 'DELETE' \
  'http://zfsdev.root.dom:5000/api/zfs/datasets/tank%2Fmyds%2FROOT?flags=1' \
  -H 'accept: text/plain'
Where the flags are the same as you would pass onto zfs (translated into a bitwise enumeration).

Of course if you want full scripting control you need to use zfs/zpool directly - but its also more work.

My plan is to have a good range of commands implemented in the library & API, but I will obviously prioritize the most used commands and leave out the commands that is used less often, like
Code:
zfs project
zfs userspace
zfs groupspace

and so forth
But there are plenty of commands to be wrapped and tested to keep me busy for a while :)

And in terms of bugfixes - its not that often that a bugfix requires that you rewrite your own code - its only with new features and possibly deprecated functionality.
 
Last edited:

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
There is a decent sized ZFS community on Reddit r/zfs
I have considered that - its definetely ZFS - I just wonder if they would be upset :) It seems like its mostly questions regarding zfs that is being posted there.

EDIT: I added a post there, lets see if they bring out the pitchforks
 
Last edited:

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
A couple of updates - api now supports creation/deletion of pools, command history for the pool, info for the pool + a couple of more utility methods, i.e. get a list of devices that can be used for a pool etc.

Bash:
curl -X 'POST' \
  'http://localhost:5000/api/zfs/pools' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "tank2",
  "vDevs": [
    {
      "type": 0,
      "devices": ["/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1","/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2"]
    }
  ]
}'
Pool creation should support all types of vdevs and special devices - also the ZFS library has gotten more modular and increased test coverage :)
The example above is just a mirror.
You can also add pool properties and default dataset properties to the same json document and the pool gets born with those properties.
 

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
An update to anyone interested.

I have fully completed the ZFS .NET library's implementation of the ZFS commands that will be part of the first version

Also every build is integration tested against unbuntu 20.04 & ubuntu 22.04, which runs zfs 0.86 & zfs 2.14

I expect to finish up the remaining zpool commands that I want to have in the first version of the library - and then I will head over to the zfs api and start using the zfs library that I have implemented - so I can finally finish up the API that I wanted to build.
 
  • Like
Reactions: zac1

zac1

Active Member
Oct 1, 2022
117
65
28
This is great — appreciate the ideas and work you've shared! I've been considering different approaches to building a database before I stumbled upon ZFS and realized it already incorporates many of the features I had in mind for the database — CoW, compression, encryption, mirroring/replication — so now I'm considering simply adding formal structures on top of ZFS. Your API might come in handy! :cool:
 

Bjorn Smith

Well-Known Member
Sep 3, 2019
684
360
63
48
r00t.dk
This is great — appreciate the ideas and work you've shared! I've been considering different approaches to building a database before I stumbled upon ZFS and realized it already incorporates many of the features I had in mind for the database — CoW, compression, encryption, mirroring/replication — so now I'm considering simply adding formal structures on top of ZFS. Your API might come in handy! :cool:
I assume you are not building on .NET since you want an API and not the library directly :)

I will update this thread when the API is "finalized" in its first form.

I will do my utmost to keep the API stable, so changes will be non-breaking.
 

zac1

Active Member
Oct 1, 2022
117
65
28
I assume you are not building on .NET since you want an API and not the library directly :)

I will update this thread when the API is "finalized" in its first form.

I will do my utmost to keep the API stable, so changes will be non-breaking.
Right, though it's a greenfield so the world is my oyster. One thing I've considered is using Cockpit's API on the backend and writing an intermediate API.