Proxmox environment feedback

Discussion in 'Linux Admins, Storage and Virtualization' started by josh, Nov 4, 2019.

  1. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    264
    Likes Received:
    41
    Migrating from Equallogic SAN to ZFS in an environment so the SAN needs a new home. Figured it's about time I experimented with proxmox so thinking of building a proxmox environment with some decomissioned C6100s.

    The idea is for a 4 node C6100 - 3 node HV, 1 node serving ZFS over iSCSI with the EQL provisioning the storage as iSCSI. Will this work? I've read that this would be the workaround to having snapshots on iSCSI backend and also allow HA and live migration since it's shared storage? What OS would I run to package the iSCSI into ZFS?
     
    #1
  2. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    Solaris and the free Solaris forks (ex OmniOS) offer the best ZFS integration, newest ZFS features and come with Comstar, an enterprise class iSCSI/FC stack, Configuring Storage Devices With COMSTAR - Oracle Solaris Administration: Devices and File Systems

    see feature set of current OmniOS stable: omniosorg/omnios-build

    ISCSi would offer snapshots and rollback, but only for the whole Lun. NFS would be the alternative. Much easier to handle and would offer filebased access and rollback. If you NFS/SMB share a ZFS filesystem, it is even possible to use Windows and SMB to copy/move/backup a VM or to restore from a snap via Windows "previous version".

    For ZFS management you can try my napp-it, a web-management tool for Solarish and with reduced features for Linux. ProxMox is now supported as a replication target (zentralized backup server) with napp-it.
     
    #2
  3. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    264
    Likes Received:
    41
    But none of these options seem to utilise the Equallogic SAN in any way
     
    #3
  4. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    As I understood, you are looking for an iSCSI storage server that provides LUNs as a replacement for the Equallogic.

    If your intention was to use the Equallogic to provide iSCSI Luns that you want to use as "disks" for a ZFS environment, you just need an Initiator on any guest ZFS OS - can be Proxmox itself. Only concern may be that in this case, ZFS has no control of real disks as you provide a remote traditional Raid or part of an array as a LUN. This would hinder ZFS to repair checksum errors (what a traditional Raid cannot detect) or guarantee secure sync writes (you must rely to a Raid-BBU on the Equallogic).

    Not sure if this works, but the best option if you want a ZFS storage server is to check if a ZFS OS can run directly on the Dell .
     
    #4
    Last edited: Nov 6, 2019
  5. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    264
    Likes Received:
    41
    But I'm relying on the SAN itself to ensure data integrity and redundancy. Just need some way for proxmox to access iscsi while still retaining snapshot functionality. If I use the iscsi initiator on proxmox I lose the snapshot functionality.
     
    #5
  6. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    If you connect the LUN on Proxmox and format it via ZFS you can create ZFS snapshots on it and do a rollback on Proxmox.

    btw
    I would call ZFS data security and redundancy superiour but this is another item.
     
    #6
  7. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    264
    Likes Received:
    41
    If I format the LUN as ZFS I would essentially be using it as local storage right? That means losing multipathing/sharing so no more live migrations and HA or am I understanding this wrong?

    I understand ZFS is the superior system but I'm trying to limit the uncertainties on the storage side of things. Otherwise a hyperconverged ceph on proxmox would probably be the better solution anyway.
     
    #7
  8. cluefr

    cluefr New Member

    Joined:
    Feb 11, 2018
    Messages:
    10
    Likes Received:
    2
    What you will advice for share zfs vm block with nappit ?
    NFS share or iSCSI or ZFS over iSCSI

    Seems Zfs over iSCSI don't use the zfs caching enable on the nappit server?

    Something like this :
    Disk <-> Nappit node <--- FC or iSCSI or ZFS-iscsi or nfs ---> Proxmox Node <-> VM on ZFS block

    Best caching for zfs would be on Proxmox node and is it usefull to do it also on nappit node ?

    Thanks
     
    #8
  9. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    From a client view ex Proxmox, a iSCSI LUN is treated like a lokal disk. Think iSCSI like a raw disk over ethernet. If you want to use your Equallogic as a SAN solution for Proxmox, no problem.

    Multipathing is a different item, is related to the Initiator and not ZFS.

    ZFS caching is done on the server that holds the ZFS Pool, Proxmox in this case. No need for a seperate ZFS server in the chain Equallogic <> Proxmox as Proxmox can provide ZFS itself (but without the advanced storage management features of a dedicated storage server).

    "Advanced" storage management on Proxmox is a cli adventure,
    see zfs(8): configures ZFS file systems - Linux man page or
    zpool(8): configures ZFS storage pools - Linux man page
     
    #9
  10. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    281
    Likes Received:
    62
    Maybe I'm a bit confused by OP. But what I gather is this: OP has Equallogic SAN he wants to reuse. In addition Several C6100s (blade?) servers he wants to reuse to be virtualization hosts with proxmox.
    What I fail to understand why ZFS is part of the equation. Yes, Proxmox could support local ZFS [ZoL?] deployment, but trying to mix Equallogic to be part of it doesn't make sense to me. My advice to keep it simple (KISS principle): Use Proxmox as the hypervisor and manage and present the storage to your proxmox hosts from your Equallogic SAN.
     
    #10
  11. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    264
    Likes Received:
    41
    Proxmox doesn't support snapshots directly with iSCSI. I noticed it does with ZFS over iSCSI. Just looking for a workaround.
     
    #11
  12. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    264
    Likes Received:
    41
    My understanding is that the storage needs to be shared for HA to work. I might be a little confused about this but I believe configuring ZFS over iSCSI would turn the storage into a local unshared volume which would no longer allow HA?
     
    #12
  13. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    Think of an iSCSI LUN like a local SAS disks. The first is a raw disk connected via network, the second via SAS. Both share the multiport/multipath feature what make them suitable for HA solutions.

    An iSCSI Lun or SAS disk itself is not a HA solution. HA is about the IT behind that is using the LUN or SAS disk.
     
    #13
  14. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    264
    Likes Received:
    41
    But if the HV with the local SAS disk went down, the guests that were stored on that drive would not be recoverable on the other HVs, thus not fulfilling HA
     
    #14
  15. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    281
    Likes Received:
    62
    OK, I did additional research and it looks like Proxmox is missing VMFS equivalent of ESXi.
    VMFS is a file system that you can place on top of raw block storage like iSCSI.
    I looked and looks and MAYBE LVM thin (supports snapshots) is possible to put on top of iSCSI storage, but honestly not sure if it worth the pain.
    Option B) is to build a large Linux machine on large iSCSI storage stored on your dell san and present it back to Proxmox VE as NFS. qcow2 will allow you to use snapshots as well but only supported on file-based storage.

    Option 3: build your own ZFS NAS using disks and shelves taken from your EqualLogic. That would be probably the most complicated route, but it also would give you most of the features and possibly performance.
     
    #15
  16. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    If you want a high available storage, "IT" must be placed between disks and service that requires the storage. Locally connected SAS disks can only give certain redundancy with multipathing not high availability.

    "IT" means a storage Cluster solution like CEPH, PaceMaker, RSF1, ZFS failover Cluster or similar that provides stable iSCSI or NFS for a guest service or Hypervisor.
     
    #16
Similar Threads: Proxmox environment
Forum Title Date
Linux Admins, Storage and Virtualization Networking Services don’t auto-start on boot up - Dell R720 w/ proxmox (Warning: Beginner User) Dec 6, 2019
Linux Admins, Storage and Virtualization Selecting a Proxmox Master Node for Cluster Oct 3, 2019
Linux Admins, Storage and Virtualization Running pfSense virtualized but serving the Proxmox as well Sep 20, 2019
Linux Admins, Storage and Virtualization Proxmox server with separate unraid storage box, is it possible? Aug 19, 2019
Linux Admins, Storage and Virtualization Proxmox issue on grub boot Jul 23, 2019

Share This Page