Proxmox ZFS pools under mergerFS, I wonder if anyone done that before ?

[vl1969]

as the subj. says, can it be done and how?
I have been planning to rebuild my home server with Proxmox VE 4.4
but as I have a single server, hardware wise, I need it to be a VM server AND a file server.
reading up on ZFS lead me to conclusion that while it is a nice robust FS it has it's issues.
for example, I have a bunch of spinners of different size in my setup.
a few 3T a few 2T a couple of 1T. to day I use OMV with BTRFS on all of them
and SnapRaid+MergerFS.

if I read correctly, I can build a ZFS pool using all of this drives by separating them into bunch of vDevs according to size and pooling the vdevs into zpool.

so as I have 4x3TB and 6x2TB and 2x1TB disks
I can do "zpool create -f tank mirror /dev/sd(3tb1) /dev/sd(3tb2) mirror /dev/sd(3tb3) /dev/sd(3tb4)
mirror /dev/sd(2tb1) /dev/sd(2tb2) ... mirror /dev/sd(2tb6) mirror /dev/sd(1tb1) /dev/sd(1tb2)"
which should give be 3+3+2+2+2+1 = 13TB zpool with 6 mirrored vdevs
NOTE: this is a file server data only. the proxmox and all vm stuff will be on zfs raid1 on a pair of 120GB or 240GB SSD (I simply do not remeber what are the system drives I use today.)
maybe I use the 1TB disks as raid1 zpool for additional system storage and backup space, if current ssds are 120GB so let say 12TB total in data space.

based on everything I read so far it seams that if I build the zfs pool as mentioned above, if I loose any of the vdevs (let say double drive failure) the whole pool is gone. that lead mt thinking that if I want a safer setup(yes I will have an external backup but as the data size expands it is a hustle to restore all), I maybe better off building a bunch of zpools out of separate vdevs and than pooling the zpools using something like mergerFS.
is it possible?

so the proposed setup would be
do
"zpool create -f tank1 mirror /dev/sd(3tb1) /dev/sd(3tb2) "
"zpool create -f tank2 mirror /dev/sd(3tb3) /dev/sd(3tb4) "
"zpool create -f tank3 mirror /dev/sd(2tb1) /dev/sd(2tb2) "
...
"zpool create -f tank5 mirror /dev/sd(2tb5) /dev/sd(2tb6) "

than make a mergerfs pool "mainData" using the tank1 through tank5

this way if any of the pools fail I only loose data on it. which at most will be 3TB or what ever the drives I use to build it. also expanding the pools will be easier as I only need to replace disk in single pool at a time or adding new pool to mergerFs volume.

PS>> is it possible to pas-through the zpool into VM , so I could setup an OMV and use mergerFS and sharing options in there. makes it easier to manage the sharing and pooling. as Proxmox already have the SMART and ZFS monitoring built in it will watch for hardware failure for me and OMV will manage the pooling of drives and sharing.

pleas state your opinions...

thanks VL

 

[MiniKnight]

Is there opportunity to sell the drives and get bigger ones? 12 drives for 26TB raw capacity is a lot to me. I'd sell them, buy 8x8TB and enjoy 30w lower power consumption and newer drives.

 

[vl1969]

not sure about it.

except for the 3TB drives, which are moderately new (under 1.5 year, in fact 2 of them are less than 6 month old)
all others are old, 3-4 years old. they seam to be good. SMART stats looks quite nice, to me at least, but I would feel bad selling them to some one if they fail. last year I lost 3TB because 2 3TB drives I bout on one of the forums just failed.
I used a BTRFS raid1 volume on raw devices using them and both fail at the same time. even though the SMART was perfect and all, they just quit. one drive was dead as a door nail, no signs of live at all and the second one was spinning but was not identifiable by any system I put it in. this are drives I collected over the last 5 years from various computers around the house and that I bought myself over the period of time. I figure I just use it until the die and I replace them gradually.
my server is a supermicro SM-SC846 24 bay box so I have space to spare. and with proper setup I will spin down drives that not used as needed. I do not really need this much of storage right now. but since I have the drives, I might as well use them.
my post is not a question for optimization sake. I just want to know if my assumptions are correct about using ZFS based pool, and if I can compensate the data safety factor by using alternative setup as described.

thanks.

 

[MiniKnight]

I've always found the simpler I get, the more reliable, especially in smaller setups. The ZFS is simple. I'd be wary of adding another layer of complexity with ZFS since every time I've seen people do that, it ends up bad. Like using HW RAID with ZFS some people don't know it's a no-no.

 

[ttabbal]

You are correct that losing both drives in a mirror will lose the pool. That said, I've never had 2 drives fail at once without any warning at all. I always start seeing errors in SMART or scrubs long before a drive goes unresponsive. I also don't run till they die, not in the main pool. The first error is a watch carefully warning. The second error is a replacement drive. If the original passes my burn in tests, it can be added to the backup server. Particularly if it's just a bad sector and the counts don't go up after testing.

The best way to avoid data loss is to treat drives like they are out to get you. Shoot first, ask questions later.

As for layering on top... I wouldn't. It's another layer that can fail. And even if both work properly individually, combined they could have a bad interaction.

If you want to use a container to manage the shares, you can bind mount filesystems into the container.

 

[vl1969]

unfortunately, I had had this happen.
year before last, I was setting up my OMV server and though I got a great deal on one of the forums.
3TB WD drives 1 yo , for 89$ each at the time it was a great deal. my data where on multiple 2TB and 1TB drives all over the place.
I got the drives, run several tests and SMART test on them all looked perfect.

build out OMV setup, added the drives to the machine, setup BTRFS raid1 pool on them
moved my data and started building the BTRFS pool on the old 2TB. so I can than use snapraid on them and used one of the 3TB as Parity.
before I had finish setting up snapraid , got error message that from OMV that one of the drives failed.
went out ,got brand new 3TB at local Microcenter for 120$, replaced the faulty drive in the pool and start sync.
after 10hours sync failed, and the drives where gone.
the new disk was virgin clean, the old drive was not accessible.
that was it, all 2.5TB of data went with it.

that is why I do not want to sell my drives since all but 3 of them are old. still work and no SMART errors but still I would feel terrible if someone got it and they break.
this was my first time when I lost 2 drives at the same time and lost data ever.


besides, I am pretty sure that MergerFS will not hurt the data.

 

[rubylaser]

unfortunately, I had had this happen.
year before last, I was setting up my OMV server and though I got a great deal on one of the forums.
3TB WD drives 1 yo , for 89$ each at the time it was a great deal. my data where on multiple 2TB and 1TB drives all over the place.
I got the drives, run several tests and SMART test on them all looked perfect.

build out OMV setup, added the drives to the machine, setup BTRFS raid1 pool on them
moved my data and started building the BTRFS pool on the old 2TB. so I can than use snapraid on them and used one of the 3TB as Parity.
before I had finish setting up snapraid , got error message that from OMV that one of the drives failed.
went out ,got brand new 3TB at local Microcenter for 120$, replaced the faulty drive in the pool and start sync.
after 10hours sync failed, and the drives where gone.
the new disk was virgin clean, the old drive was not accessible.
that was it, all 2.5TB of data went with it.

that is why I do not want to sell my drives since all but 3 of them are old. still work and no SMART errors but still I would feel terrible if someone got it and they break.
this was my first time when I lost 2 drives at the same time and lost data ever.


besides, I am pretty sure that MergerFS will not hurt the data.

I love MergerFS and use it on two bulk arrays. That being said, I would not use it in Proxmox for VM disks. I would make a single ZFS pool made up of the mirrored vdevs as you have shown above for your VM disks. Or, better yet, use SSDs for you VM disks and then use the spinners for you bulk media.

 

[vl1969]

No,no,no you misunderstood me. I am not planning to use the mergerfs for my vm disk.
The plan is to
A. Build out a proxmox box using 2xssd for system and maybe a 2x1tb spinners for local store, all vm config and os drives will be on ssd ,and all is and other images will be on second zfs pool of mirrored 1tb spinners.

B. Setup a zfs or btrfs mirror pools with other drives as I described in previous post to hold my actual data.
Now if I want to easier manage all that and maybe use mergerfs on only this data pool but if I want a guide to manage it I will need to pass this pools into an omv or freenas vm only as a data disks not os. Proxmox will have nothing to do with this disks at all.


Sent from my SM-N910T using Tapatalk

 

[rubylaser]

No,no,no you misunderstood me. I am not planning to use the mergerfs for my vm disk.
The plan is to
A. Build out a proxmox box using 2xssd for system and maybe a 2x1tb spinners for local store, all vm config and os drives will be on ssd ,and all is and other images will be on second zfs pool of mirrored 1tb spinners.

B. Setup a zfs or btrfs mirror pools with other drives as I described in previous post to hold my actual data.
Now if I want to easier manage all that and maybe use mergerfs on only this data pool but if I want a guide to manage it I will need to pass this pools into an omv or freenas vm only as a data disks not os. Proxmox will have nothing to do with this disks at all.


Sent from my SM-N910T using Tapatalk

If that's all you want. I'd just run ZFS on root with as many SSDs as you need Vms, and then use SnapRAID + mergerfs for your bulk media (I'd use two parity levels). You will end up with more available space have a very flexible way add add disks in the future while still having checksumming. Just my two cents, but that is exactly how I use one of my Proxmox hosts.

 

[vl1969]

If that's all you want. I'd just run ZFS on root with as many SSDs as you need Vms, and then use SnapRAID + mergerfs for your bulk media (I'd use two parity levels). You will end up with more available space have a very flexible way add add disks in the future while still having checksumming. Just my two cents, but that is exactly how I use one of my Proxmox hosts.

you lost me.
do you mean I need to have an SSD for each VM I want to run?

 

[Kybber]

He probably meant to say "as much SSD capacity as you need for VMs".

In my setup I run Proxmox from (mirrored) SSDs, and then I have 4*4TB as Snapraid + MergerFS for media and 4*4TB as ZFS (2*mirror) for anything else, including VMs that don't fit on the SSDs (e.g. a 300GB+ disk for Steam games connected to a Windows-VM).

 

[rubylaser]

you lost me.
do you mean I need to have an SSD for each VM I want to run?

No. You build a ZFS pool out of SSDs to run your VMs. I have (6) Intel S3700's setup in (3) mirrored vdevs that all my virtual machines run off of.

 

[vl1969]

He probably meant to say "as much SSD capacity as you need for VMs".

In my setup I run Proxmox from (mirrored) SSDs, and then I have 4*4TB as Snapraid + MergerFS for media and 4*4TB as ZFS (2*mirror) for anything else, including VMs that don't fit on the SSDs (e.g. a 300GB+ disk for Steam games connected to a Windows-VM).

oh, ok, that makes more sense

so my basic setup should be ok.
Get the Proxmox on 2x120GB SSD in ZFS raid-1 setup (not raidz or anything) basic mirrored rpool.
Add second ZFS mirrored ldpool (ld- is for "local data") using 2x1TB spinners. I have them might as well use them. I am pretty sure it will be more than adequate for my needs.
in my test setup (using Hyper-v VM for my proxmox install and test) I have 127GB disk that shows me that Proxmox install uses about 1.5GB

root@pve1:~# zfs list
NAME                           USED  AVAIL  REFER  MOUNTPOINT
rpool                         6.21G   116G    96K  /rpool
rpool/ROOT                    1.45G   116G    96K  /rpool/ROOT
rpool/ROOT/pve-1              1.45G   116G  1.45G  /
rpool/data                     522M   116G    96K  /rpool/data
rpool/data/subvol-100-disk-1   522M  7.49G   522M  /rpool/data/subvol-100-disk-1
rpool/swap                    4.25G   120G    64K  -
tank                           276K   289G    96K  /tank

most VMs do not take up much of a space if you only run OS on them.
so I can probably fit a bunch VMs on SSD main rpool/data local store, keep the ISO and other images on
ldpool, as well as any big VMs

and use the rest disks 2 and 3 TB spinners to hold my other data. managing and sharing them via OMV VM and NFS shares from Proxmox. maybe as planned load a WebMin to help manage NFS/SAMBA shares on HOST. since Proxmox does not have this services it will not interfere.

now, may I ask what/how do you use the drives with snapraid and mergerfs ?
I understand you use ZFS on the Main drives like Os and local store, BUT
what File System you use on your data drives?
how do you manage them?
please share

 

[Kybber]

I've actually followed @rubylaser's guides pretty much to the point (thanks, @rubylaser !)
Setting up SnapRAID on Ubuntu to Create a Flexible Home Media Fileserver - Zack Reed - Design and Coding
Mergerfs - another good option to pool your SnapRAID disks - Zack Reed - Design and Coding
Script for Nightly SnapRAID Syncs and Scrubs - Zack Reed - Design and Coding

Regarding sharing directories via NFS/Samba, you may want to have a look at the Turnkey File Server. It is available as a container in Proxmox. Since it's a container, you can bind-mount the directories and use Webmin to configure sharing. I set it up just a couple of days ago - very quick and painless.

 

[vl1969]

I've actually followed @rubylaser's guides pretty much to the point (thanks, @rubylaser !)
Setting up SnapRAID on Ubuntu to Create a Flexible Home Media Fileserver - Zack Reed - Design and Coding
Mergerfs - another good option to pool your SnapRAID disks - Zack Reed - Design and Coding
Script for Nightly SnapRAID Syncs and Scrubs - Zack Reed - Design and Coding

Regarding sharing directories via NFS/Samba, you may want to have a look at the Turnkey File Server. It is available as a container in Proxmox. Since it's a container, you can bind-mount the directories and use Webmin to configure sharing. I set it up just a couple of days ago - very quick and painless.

yes I have been looking at this for a bit as well.
my main problem is that it does not work on my nested VM setup. my VMs can not get to network. hence can not get proper IP. so I can not try to full setup. have to wait until I can get my main hardware server build up to test this. thanks. been very busy lately.