ZFS on Proxmox host, VM, or Dedicated?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Thomas H

Member
Dec 2, 2017
60
25
18
53
System Configuration:
  • 10 * 4TB HGST HDD
  • ZFS RAIDZ2 using HBA adapter in IT mode
  • Two systems available: Xeon 1245/1270 v3 servers with 10GbE

Hi, rebuilding homelab and need advice on how to approach ZFS storage solution. It will be used for general file share, streaming media, web development, video surveillance, etc. Uptime is important so don't want storage to go down when host is being restarted or crashed.

Which is best for main ZFS pool?
  • ZFS on a dedicated system. Running FreeNAS, GUI managed, NAS only.
  • ZFS on proxmox host. Native ZFS on host, CLI managed, hyper-converged virtualization.
  • ZFS inside a VM on proxmox host. VM running FreeNAS with PCI passthrough of HBA adapter in IT mode.
  • Other ideas?
On #2, I am worried if host is restart/down will bring all services down. Also, ZFS taking away RAM from host.
On #1 and #3, if host or FreeNAS VM dies, I can quickly rebuild and restore access to data.
 
  • Like
Reactions: RandyC

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
What else besides a ZFS Host do you need /run?
You could use FreeNas as Host and run VMs in bhyve ...
 

RandyC

Member
Mar 1, 2014
70
15
8
Portland, OR
How about having both systems run proxmox and setup ceph storage?

I have ZFS running on proxmox through CLI setup. It does take away memory for running VMs, so I am very interested in this scenario, and possible alternatives.

I know I had to limit the ZFS RAM cache size to 8GB, otherwise it used 16GB of my 32GB proxmox system RAM.
 

groove

Member
Sep 21, 2011
90
31
18
I put zfs storage on it’s own host and made it available via SRP (to an ESXi host) and via iSer (Solaris host running zones). Still in testing but performace is great and I don’t loose any memory on the hosts running my vms.
 
  • Like
Reactions: gigatexal

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
I put zfs storage on it’s own host and made it available via SRP (to an ESXi host) and via iSer (Solaris host running zones). Still in testing but performace is great and I don’t loose any memory on the hosts running my vms.
i wonder how you access the srp target from ESXi .. looking to do so for a poc, too.
 

Thomas H

Member
Dec 2, 2017
60
25
18
53
What else besides a ZFS Host do you need /run?
You could use FreeNas as Host and run VMs in bhyve ...
VMs running include pfSense, Windows Server, WordPress LAMP, Plex, UniFi controller, ownCloud, FreePBX, Home Assistant, testing OSes/distros, etc. I've been testing Hyper-V Server 2016, esxi v6.5, and proxmox. Leaning towards proxmox as it including LXC, ZFS, and other enterprise features.

How about having both systems run proxmox and setup ceph storage?

I have ZFS running on proxmox through CLI setup. It does take away memory for running VMs, so I am very interested in this scenario, and possible alternatives.

I know I had to limit the ZFS RAM cache size to 8GB, otherwise it used 16GB of my 32GB proxmox system RAM.
I thought you need a minimum of three nodes for ceph?

I put zfs storage on it’s own host and made it available via SRP (to an ESXi host) and via iSer (Solaris host running zones). Still in testing but performace is great and I don’t loose any memory on the hosts running my vms.
What is your host? Currently, I am testing one with NAS4Free (in a VM) but thinking of switching to FreeNAS (on dedicated system to avoid passthrough issues).
 

groove

Member
Sep 21, 2011
90
31
18
My ‘san’ is a solaris 11.3 host that runs on a xeon E3-1230 v2 / 32 GB ECC box. It is equipped with a ConnectX-3 card. I have a infiniband setup with a IS3050 infiniband switch that I scored for 150 hucks on fleabay. I’m getting ~ 30 gbps iperf bandwidth between 2 solaris host. My esxi box is currently down so don’t have the stats for that yet.