Custom storage plugins for Proxmox

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
hi,
do you have any lun's on your scst ?

the plugin should create lun's on also created zvols/datasets if you add volumes via proxmox.
Therefore an Target with empty lun 0 needs to be present and confgured in storage.cfg

didn't use scstadmin as it's just one more thing between the plugin and scst sysfs that can break/change ;) (guess scstadmin is just a cli for sysfs in the end)
imho the sysfs is more stable and also clear to use.

can have a closer look/help more from monday on, will be skiing for some days.
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
@nephri how are you doing with FreeNAS ? have you put it in production?
i have scst Running on my playground-Cluster and also asked for inclusion of parts of the changes in pve on the devel-list. this should be possible if it's reasonable. will need to setup a proper perl/pve Development environment and re-work parts to supply clean patch-series. just haven't found enough time as my regular work was quite demanding during the last weeks :(
but will definitely come back to this with a day or two to finish/cleanup and submitt patches for qemu-server for clones and the main zfs-plugin to support cloning and auth.
 

dbo

New Member
Feb 22, 2017
2
0
1
54
Yes, in fact there are already 16 luns. They are used by iscsi virtio disks devices on my pve servers. As it is a production server, I don't want to break it, so I didn't try creating new luns. Will test on another target...

In fact scstadmin is a little more than a simple cli. It has some logic inside. My first tought was to avoid duplicating it, but maybe I'm wrong :)
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
yes, there is somewhere a scst Perl Lib (locate / find should get it quite fast), that i guess is used by scstadmin. that would be really handy to use ;)

i would absolutely not use it on a box that has production volumes, as it might create duplicate datasets. i thought about putting another production host on my playground and then saw the urgent need to prefix created volumes with the Cluster-Name or something unique ;)

for trial, its maybe best to setup a vm with scst and watch what happens there / what datasets on zfs and devices/lun's in scst are created.
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
Hi Filip,
and welcome to the STH Forums.

Regarding your Question, I'm not currently planning this.
Basically, i gave up on FreeNAS as i dont't use it and went into interesting issues that resulted in the VM's going ro when the config of FreeNAS is changed via API. Guess this is because the target is restarted and not reloaded in most of the cases.

What i use in lab's currently is SCST on ESOS as a target, this works quite well, and if i find time i do some improvement on it, too.
One 'problem' is that a small change in QemuServer.pm would be necessary to be able to clone from a snapshot, what is a quite cool feature for doing backups. (snapshot running, create a clone from this snap, backup the clone or use it for diagnostics)

I just don't have the time to prepare proper patches that would maybe be accepted.
Also some parts are still missing, like resizing, defining peers of ZFS arrays that could do send/receive between each other (that would mean: create a clone from Snapshot on Array A on Array B) and most important IB-Support.
Another fact is that imho it's currently impossible to inject dialogs into the pve gui from custom storage plugins, what would be nice.

If you're interested in my work against SCST so far feel free to PM, i'm ok with sharing and giving some assistance with setup. But be aware that i'd currently not encourage anyone to use this in production ;)

Alex

Edit: i guess FreeNAS might be integrated by proxmox in one of the next releases, if i remember right from skimming the mailing list. So if it's about FreeNAS you could find something there.
 
Last edited:

EluRex

Active Member
Apr 28, 2015
218
78
28
Los Angeles, CA
Hi Filip,
and welcome to the STH Forums.

Regarding your Question, I'm not currently planning this.
Dear alex,

I am using scst currently on proxmox itself wtih IB, however, all settings are done via scstadmin cli and also srp initiator only.

Having a web gui would be great but it will stop me from using SCST as it is far better and stable than other target out there.
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
Hi, not sure what you want/need ..
Do you run a zvol and export this via srp ?

With ib_srp over redundant paths i ended up putting lvm on it and marking the storage as shared.
 

hellrider8881

New Member
Jul 31, 2018
2
0
1
Hi _alex,
did you ever get to the MPIO implementation? I am currently using official ZFS over ISCSI plugin with OmniOS on storage side, however I am forced to use LACP as there is no official MPIO support yet (still) which greatly reduces the performance per VM and this is the only thing that still prevents us to use the plugin on in the real environment. I have actually wanted to implement this by modifying the official plugin (I know the updates would break this and stuff..) but my Perl skills are, well, not sufficient for this task shall we say :)

Thanks!
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
Hey,
no, i haven't followed this much more, besides 'maintaining' this PlugIn for possible future use.
Afaik, MPIO on the ZFS-Over-ISCSI Plugin is PITA.
ISCSI is handled by libiscsi in KVM, so KVM itself would need to handle MPIO, what is quite unlikely to happen.
afaik Proxmox itself doesn't care about MPIO at all ..
(also, libiscsi compiled into KVM in Proxomox is that old that it can't do iser and proxmox-devs obviously don't want to compile KVM with newer libiscsi - what is more than sad)

Other option would be mpio on the OS-Side and using the local mpio-device as storage. This works (i do this with SRP-Targets) but doesn't interact at all with zfs over iscsi. So a lot (if not too many) changes would need to be done on mics parts to make this happen.
You might be able to export a ZVOL, configure MPIO and put LVM on top of the device and go with a LVM-Based storage where you tick the 'shared' checkbox. But this involves loosing some goodies ZFS would offer.
 

hellrider8881

New Member
Jul 31, 2018
2
0
1
That's a shame, as ZFS over ISCSI is really good solution for small setups with single storage server and it would be really neat to have MPIO as well. At least I now know there is more to it than simple "add some Perl glue to make it work". Maybe next time.

Update: When posting the original comment, I actually had in mind a solution similar to the official Proxmox ISCSI plugin which mounts a block device locally on Proxmox node and is also able to use MPIO. From technical standpoint I see no reason why this shouldn't be possible with ZoI, only instead of using iscsi target (ip address) directly with KVM, the MPIO device would be used. The only "issue" here is that the Proxmox cluster nodes would need to know which MPIO device is used by which VM, however this should not be problematic as the official ISCSI plugin uses MPIO devices too and is also able to use them as Shared storage. As far as I am aware the ZoI plugin actually only creates and exports necessary LUNs on storage machine, although in that case it may be easier to just create a custom plugin based on the official one.
 
Last edited:

nephri

Active Member
Sep 23, 2015
541
106
43
45
Paris, France
I reply to an old thread

But i was playing again with freenas + proxmox, so i read again what i wrote few years back and posted on this thread.
In the meantime, i checked if it is always needed and if Proxmox didn't included some native stuffs like this for doing ISCSI with freenas.

What a surprise when i found a git repository TheGrandWazoo/freenas-proxmox that do quite the same stuff.
I decided to read how it achieve some stuffs (for seeing how deeply the integration is) and i found that it use a large code base of what i posted here !

Since, he did some cleanup / completion / compatibility changes.

i thinked this code quite dead and finally it found a public area.
It's just sad i didn't set any licence mention in order to have at least a mention of the initial code source.
 
Last edited: