Proxmox environment feedback

josh

Active Member
Oct 21, 2013
310
87
28
Migrating from Equallogic SAN to ZFS in an environment so the SAN needs a new home. Figured it's about time I experimented with proxmox so thinking of building a proxmox environment with some decomissioned C6100s.

The idea is for a 4 node C6100 - 3 node HV, 1 node serving ZFS over iSCSI with the EQL provisioning the storage as iSCSI. Will this work? I've read that this would be the workaround to having snapshots on iSCSI backend and also allow HA and live migration since it's shared storage? What OS would I run to package the iSCSI into ZFS?
 

gea

Well-Known Member
Dec 31, 2010
2,485
837
113
DE
Solaris and the free Solaris forks (ex OmniOS) offer the best ZFS integration, newest ZFS features and come with Comstar, an enterprise class iSCSI/FC stack, Configuring Storage Devices With COMSTAR - Oracle Solaris Administration: Devices and File Systems

see feature set of current OmniOS stable: omniosorg/omnios-build

ISCSi would offer snapshots and rollback, but only for the whole Lun. NFS would be the alternative. Much easier to handle and would offer filebased access and rollback. If you NFS/SMB share a ZFS filesystem, it is even possible to use Windows and SMB to copy/move/backup a VM or to restore from a snap via Windows "previous version".

For ZFS management you can try my napp-it, a web-management tool for Solarish and with reduced features for Linux. ProxMox is now supported as a replication target (zentralized backup server) with napp-it.
 

josh

Active Member
Oct 21, 2013
310
87
28
Solaris and the free Solaris forks (ex OmniOS) offer the best ZFS integration, newest ZFS features and come with Comstar, an enterprise class iSCSI/FC stack, Configuring Storage Devices With COMSTAR - Oracle Solaris Administration: Devices and File Systems

see feature set of current OmniOS stable: omniosorg/omnios-build

ISCSi would offer snapshots and rollback, but only for the whole Lun. NFS would be the alternative. Much easier to handle and would offer filebased access and rollback. If you NFS/SMB share a ZFS filesystem, it is even possible to use Windows and SMB to copy/move/backup a VM or to restore from a snap via Windows "previous version".

For ZFS management you can try my napp-it, a web-management tool for Solarish and with reduced features for Linux. ProxMox is now supported as a replication target (zentralized backup server) with napp-it.
But none of these options seem to utilise the Equallogic SAN in any way
 

gea

Well-Known Member
Dec 31, 2010
2,485
837
113
DE
As I understood, you are looking for an iSCSI storage server that provides LUNs as a replacement for the Equallogic.

If your intention was to use the Equallogic to provide iSCSI Luns that you want to use as "disks" for a ZFS environment, you just need an Initiator on any guest ZFS OS - can be Proxmox itself. Only concern may be that in this case, ZFS has no control of real disks as you provide a remote traditional Raid or part of an array as a LUN. This would hinder ZFS to repair checksum errors (what a traditional Raid cannot detect) or guarantee secure sync writes (you must rely to a Raid-BBU on the Equallogic).

Not sure if this works, but the best option if you want a ZFS storage server is to check if a ZFS OS can run directly on the Dell .
 
Last edited:

josh

Active Member
Oct 21, 2013
310
87
28
As I understood, you are looking for an iSCSI storage server that provides LUNs as a replacement for the Equallogic.

If your intention was to use the Equallogic to provide iSCSI Luns that you want to use as "disks" for a ZFS environment, you just need an Initiator on any guest ZFS OS - can be Proxmox itself. Only concern may be that in this case, ZFS has no control of real disks as you provide a remote traditional Raid or part of an array as a LUN. This would hinder ZFS to repair checksum errors (what a traditional Raid cannot detect) or guarantee secure sync writes (you must rely to a Raid-BBU on the Equallogic).
But I'm relying on the SAN itself to ensure data integrity and redundancy. Just need some way for proxmox to access iscsi while still retaining snapshot functionality. If I use the iscsi initiator on proxmox I lose the snapshot functionality.
 

gea

Well-Known Member
Dec 31, 2010
2,485
837
113
DE
If you connect the LUN on Proxmox and format it via ZFS you can create ZFS snapshots on it and do a rollback on Proxmox.

btw
I would call ZFS data security and redundancy superiour but this is another item.
 

josh

Active Member
Oct 21, 2013
310
87
28
If you connect the LUN on Proxmox and format it via ZFS you can create ZFS snapshots on it and do a rollback on Proxmox.

btw
I would call ZFS data security and redundancy superiour but this is another item.
If I format the LUN as ZFS I would essentially be using it as local storage right? That means losing multipathing/sharing so no more live migrations and HA or am I understanding this wrong?

I understand ZFS is the superior system but I'm trying to limit the uncertainties on the storage side of things. Otherwise a hyperconverged ceph on proxmox would probably be the better solution anyway.
 

cluefr

New Member
Feb 11, 2018
10
2
3
45
Solaris and the free Solaris forks (ex OmniOS) offer the best ZFS integration, newest ZFS features and come with Comstar, an enterprise class iSCSI/FC stack, Configuring Storage Devices With COMSTAR - Oracle Solaris Administration: Devices and File Systems

see feature set of current OmniOS stable: omniosorg/omnios-build

ISCSi would offer snapshots and rollback, but only for the whole Lun. NFS would be the alternative. Much easier to handle and would offer filebased access and rollback. If you NFS/SMB share a ZFS filesystem, it is even possible to use Windows and SMB to copy/move/backup a VM or to restore from a snap via Windows "previous version".

For ZFS management you can try my napp-it, a web-management tool for Solarish and with reduced features for Linux. ProxMox is now supported as a replication target (zentralized backup server) with napp-it.
What you will advice for share zfs vm block with nappit ?
NFS share or iSCSI or ZFS over iSCSI

Seems Zfs over iSCSI don't use the zfs caching enable on the nappit server?

Something like this :
Disk <-> Nappit node <--- FC or iSCSI or ZFS-iscsi or nfs ---> Proxmox Node <-> VM on ZFS block

Best caching for zfs would be on Proxmox node and is it usefull to do it also on nappit node ?

Thanks
 

gea

Well-Known Member
Dec 31, 2010
2,485
837
113
DE
From a client view ex Proxmox, a iSCSI LUN is treated like a lokal disk. Think iSCSI like a raw disk over ethernet. If you want to use your Equallogic as a SAN solution for Proxmox, no problem.

Multipathing is a different item, is related to the Initiator and not ZFS.

ZFS caching is done on the server that holds the ZFS Pool, Proxmox in this case. No need for a seperate ZFS server in the chain Equallogic <> Proxmox as Proxmox can provide ZFS itself (but without the advanced storage management features of a dedicated storage server).

"Advanced" storage management on Proxmox is a cli adventure,
see zfs(8): configures ZFS file systems - Linux man page or
zpool(8): configures ZFS storage pools - Linux man page
 

BoredSysadmin

Active Member
Mar 2, 2019
407
113
43
Maybe I'm a bit confused by OP. But what I gather is this: OP has Equallogic SAN he wants to reuse. In addition Several C6100s (blade?) servers he wants to reuse to be virtualization hosts with proxmox.
What I fail to understand why ZFS is part of the equation. Yes, Proxmox could support local ZFS [ZoL?] deployment, but trying to mix Equallogic to be part of it doesn't make sense to me. My advice to keep it simple (KISS principle): Use Proxmox as the hypervisor and manage and present the storage to your proxmox hosts from your Equallogic SAN.
 

josh

Active Member
Oct 21, 2013
310
87
28
Maybe I'm a bit confused by OP. But what I gather is this: OP has Equallogic SAN he wants to reuse. In addition Several C6100s (blade?) servers he wants to reuse to be virtualization hosts with proxmox.
What I fail to understand why ZFS is part of the equation. Yes, Proxmox could support local ZFS [ZoL?] deployment, but trying to mix Equallogic to be part of it doesn't make sense to me. My advice to keep it simple (KISS principle): Use Proxmox as the hypervisor and manage and present the storage to your proxmox hosts from your Equallogic SAN.
Proxmox doesn't support snapshots directly with iSCSI. I noticed it does with ZFS over iSCSI. Just looking for a workaround.
 

josh

Active Member
Oct 21, 2013
310
87
28
From a client view ex Proxmox, a iSCSI LUN is treated like a lokal disk. Think iSCSI like a raw disk over ethernet. If you want to use your Equallogic as a SAN solution for Proxmox, no problem.

Multipathing is a different item, is related to the Initiator and not ZFS.

ZFS caching is done on the server that holds the ZFS Pool, Proxmox in this case. No need for a seperate ZFS server in the chain Equallogic <> Proxmox as Proxmox can provide ZFS itself (but without the advanced storage management features of a dedicated storage server).

"Advanced" storage management on Proxmox is a cli adventure,
see zfs(8): configures ZFS file systems - Linux man page or
zpool(8): configures ZFS storage pools - Linux man page
My understanding is that the storage needs to be shared for HA to work. I might be a little confused about this but I believe configuring ZFS over iSCSI would turn the storage into a local unshared volume which would no longer allow HA?
 

gea

Well-Known Member
Dec 31, 2010
2,485
837
113
DE
Think of an iSCSI LUN like a local SAS disks. The first is a raw disk connected via network, the second via SAS. Both share the multiport/multipath feature what make them suitable for HA solutions.

An iSCSI Lun or SAS disk itself is not a HA solution. HA is about the IT behind that is using the LUN or SAS disk.
 

josh

Active Member
Oct 21, 2013
310
87
28
Think of an iSCSI LUN like a local SAS disks. The first is a raw disk connected via network, the second via SAS. Both share the multiport/multipath feature what make them suitable for HA solutions.

An iSCSI Lun or SAS disk itself is not a HA solution. HA is about the IT behind that is using the LUN or SAS disk.
But if the HV with the local SAS disk went down, the guests that were stored on that drive would not be recoverable on the other HVs, thus not fulfilling HA
 

BoredSysadmin

Active Member
Mar 2, 2019
407
113
43
Proxmox doesn't support snapshots directly with iSCSI. I noticed it does with ZFS over iSCSI. Just looking for a workaround.
OK, I did additional research and it looks like Proxmox is missing VMFS equivalent of ESXi.
VMFS is a file system that you can place on top of raw block storage like iSCSI.
I looked and looks and MAYBE LVM thin (supports snapshots) is possible to put on top of iSCSI storage, but honestly not sure if it worth the pain.
Option B) is to build a large Linux machine on large iSCSI storage stored on your dell san and present it back to Proxmox VE as NFS. qcow2 will allow you to use snapshots as well but only supported on file-based storage.

Option 3: build your own ZFS NAS using disks and shelves taken from your EqualLogic. That would be probably the most complicated route, but it also would give you most of the features and possibly performance.
 

gea

Well-Known Member
Dec 31, 2010
2,485
837
113
DE
But if the HV with the local SAS disk went down, the guests that were stored on that drive would not be recoverable on the other HVs, thus not fulfilling HA
If you want a high available storage, "IT" must be placed between disks and service that requires the storage. Locally connected SAS disks can only give certain redundancy with multipathing not high availability.

"IT" means a storage Cluster solution like CEPH, PaceMaker, RSF1, ZFS failover Cluster or similar that provides stable iSCSI or NFS for a guest service or Hypervisor.