Custom storage plugins for Proxmox

Discussion in 'Linux Admins, Storage and Virtualization' started by _alex, Dec 28, 2016.

  1. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    848
    Likes Received:
    88
    hi,
    do you have any lun's on your scst ?

    the plugin should create lun's on also created zvols/datasets if you add volumes via proxmox.
    Therefore an Target with empty lun 0 needs to be present and confgured in storage.cfg

    didn't use scstadmin as it's just one more thing between the plugin and scst sysfs that can break/change ;) (guess scstadmin is just a cli for sysfs in the end)
    imho the sysfs is more stable and also clear to use.

    can have a closer look/help more from monday on, will be skiing for some days.
     
    #61
  2. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    848
    Likes Received:
    88
    @nephri how are you doing with FreeNAS ? have you put it in production?
    i have scst Running on my playground-Cluster and also asked for inclusion of parts of the changes in pve on the devel-list. this should be possible if it's reasonable. will need to setup a proper perl/pve Development environment and re-work parts to supply clean patch-series. just haven't found enough time as my regular work was quite demanding during the last weeks :(
    but will definitely come back to this with a day or two to finish/cleanup and submitt patches for qemu-server for clones and the main zfs-plugin to support cloning and auth.
     
    #62
  3. dbo

    dbo New Member

    Joined:
    Feb 22, 2017
    Messages:
    2
    Likes Received:
    0
    Yes, in fact there are already 16 luns. They are used by iscsi virtio disks devices on my pve servers. As it is a production server, I don't want to break it, so I didn't try creating new luns. Will test on another target...

    In fact scstadmin is a little more than a simple cli. It has some logic inside. My first tought was to avoid duplicating it, but maybe I'm wrong :)
     
    #63
  4. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    848
    Likes Received:
    88
    yes, there is somewhere a scst Perl Lib (locate / find should get it quite fast), that i guess is used by scstadmin. that would be really handy to use ;)

    i would absolutely not use it on a box that has production volumes, as it might create duplicate datasets. i thought about putting another production host on my playground and then saw the urgent need to prefix created volumes with the Cluster-Name or something unique ;)

    for trial, its maybe best to setup a vm with scst and watch what happens there / what datasets on zfs and devices/lun's in scst are created.
     
    #64
  5. Filip Van Ham

    Filip Van Ham New Member

    Joined:
    Jan 22, 2017
    Messages:
    1
    Likes Received:
    0
    Any updates on including this in pve releases?
     
    #65
  6. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    848
    Likes Received:
    88
    Hi Filip,
    and welcome to the STH Forums.

    Regarding your Question, I'm not currently planning this.
    Basically, i gave up on FreeNAS as i dont't use it and went into interesting issues that resulted in the VM's going ro when the config of FreeNAS is changed via API. Guess this is because the target is restarted and not reloaded in most of the cases.

    What i use in lab's currently is SCST on ESOS as a target, this works quite well, and if i find time i do some improvement on it, too.
    One 'problem' is that a small change in QemuServer.pm would be necessary to be able to clone from a snapshot, what is a quite cool feature for doing backups. (snapshot running, create a clone from this snap, backup the clone or use it for diagnostics)

    I just don't have the time to prepare proper patches that would maybe be accepted.
    Also some parts are still missing, like resizing, defining peers of ZFS arrays that could do send/receive between each other (that would mean: create a clone from Snapshot on Array A on Array B) and most important IB-Support.
    Another fact is that imho it's currently impossible to inject dialogs into the pve gui from custom storage plugins, what would be nice.

    If you're interested in my work against SCST so far feel free to PM, i'm ok with sharing and giving some assistance with setup. But be aware that i'd currently not encourage anyone to use this in production ;)

    Alex

    Edit: i guess FreeNAS might be integrated by proxmox in one of the next releases, if i remember right from skimming the mailing list. So if it's about FreeNAS you could find something there.
     
    #66
    Last edited: Jul 18, 2017
  7. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    202
    Likes Received:
    69
    Dear alex,

    I am using scst currently on proxmox itself wtih IB, however, all settings are done via scstadmin cli and also srp initiator only.

    Having a web gui would be great but it will stop me from using SCST as it is far better and stable than other target out there.
     
    #67
  8. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    848
    Likes Received:
    88
    Hi, not sure what you want/need ..
    Do you run a zvol and export this via srp ?

    With ib_srp over redundant paths i ended up putting lvm on it and marking the storage as shared.
     
    #68
  9. hellrider8881

    hellrider8881 New Member

    Joined:
    Jul 31, 2018
    Messages:
    2
    Likes Received:
    0
    Hi _alex,
    did you ever get to the MPIO implementation? I am currently using official ZFS over ISCSI plugin with OmniOS on storage side, however I am forced to use LACP as there is no official MPIO support yet (still) which greatly reduces the performance per VM and this is the only thing that still prevents us to use the plugin on in the real environment. I have actually wanted to implement this by modifying the official plugin (I know the updates would break this and stuff..) but my Perl skills are, well, not sufficient for this task shall we say :)

    Thanks!
     
    #69
  10. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    848
    Likes Received:
    88
    Hey,
    no, i haven't followed this much more, besides 'maintaining' this PlugIn for possible future use.
    Afaik, MPIO on the ZFS-Over-ISCSI Plugin is PITA.
    ISCSI is handled by libiscsi in KVM, so KVM itself would need to handle MPIO, what is quite unlikely to happen.
    afaik Proxmox itself doesn't care about MPIO at all ..
    (also, libiscsi compiled into KVM in Proxomox is that old that it can't do iser and proxmox-devs obviously don't want to compile KVM with newer libiscsi - what is more than sad)

    Other option would be mpio on the OS-Side and using the local mpio-device as storage. This works (i do this with SRP-Targets) but doesn't interact at all with zfs over iscsi. So a lot (if not too many) changes would need to be done on mics parts to make this happen.
    You might be able to export a ZVOL, configure MPIO and put LVM on top of the device and go with a LVM-Based storage where you tick the 'shared' checkbox. But this involves loosing some goodies ZFS would offer.
     
    #70
  11. hellrider8881

    hellrider8881 New Member

    Joined:
    Jul 31, 2018
    Messages:
    2
    Likes Received:
    0
    That's a shame, as ZFS over ISCSI is really good solution for small setups with single storage server and it would be really neat to have MPIO as well. At least I now know there is more to it than simple "add some Perl glue to make it work". Maybe next time.

    Update: When posting the original comment, I actually had in mind a solution similar to the official Proxmox ISCSI plugin which mounts a block device locally on Proxmox node and is also able to use MPIO. From technical standpoint I see no reason why this shouldn't be possible with ZoI, only instead of using iscsi target (ip address) directly with KVM, the MPIO device would be used. The only "issue" here is that the Proxmox cluster nodes would need to know which MPIO device is used by which VM, however this should not be problematic as the official ISCSI plugin uses MPIO devices too and is also able to use them as Shared storage. As far as I am aware the ZoI plugin actually only creates and exports necessary LUNs on storage machine, although in that case it may be easier to just create a custom plugin based on the official one.
     
    #71
    Last edited: Aug 1, 2018
Similar Threads: Custom storage
Forum Title Date
Linux Admins, Storage and Virtualization high capacity, high throughput storage platform Mar 26, 2018
Linux Admins, Storage and Virtualization Proxmox / ZFS Storage Configuration Jan 27, 2018
Linux Admins, Storage and Virtualization ZFS on Linux perf tuning for all SSD storage guide? Jan 12, 2018
Linux Admins, Storage and Virtualization Switching from Windows based Storage/Transcode Server to Ubuntu Advice Nov 21, 2017
Linux Admins, Storage and Virtualization Proxmox Blade Cluster storage options Aug 20, 2017

Share This Page