Storage Server w/Mergerfs + Snapraid + Proxmox + ZFS??

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Marsh

Moderator
May 12, 2013
2,643
1,496
113
I tar backup the config files in /etc , docker container directory
Nightly backup to a HD within the same host, also to an external fileserver as well.


My media clients are Kodi based, using Emby for Kodi.

The Emby server is for updating Kodi internal database via Emby for Kodi plugin.

I could watched all my shows without Emby container running.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
I tar backup the config files in /etc , docker container directory
Nightly backup to a HD within the same host, also to an external fileserver as well.


My media clients are Kodi based, using Emby for Kodi.

The Emby server is for updating Kodi internal database via Emby for Kodi plugin.

I could watched all my shows without Emby container running.
Ah interesting. I didn't know one could simply backup the /etc directory and simply restore that on a fresh install. That is good to know. Care to share anything about your ansible playbook? I plan to get that going if this build takes off soon because I want as seamless of an install process as I can get.
 

Marsh

Moderator
May 12, 2013
2,643
1,496
113
Make a shell script , run the shutdown-container and shell script using cron , run it nightly

# Config files backup directory
config_bk=$HOME/BLD_`hostname`/config_bk
BKDIR=backupfile-director

cd $HOME

mkdir -p $config_bk/etc/
mkdir -p $config_bk/etc/samba
mkdir -p $config_bk/etc/netplan

cp -p /etc/hosts $config_bk/etc/
cp -p /etc/exports $config_bk/etc/
cp -p /etc/environment $config_bk/etc/
cp -p /etc/fstab* $config_bk/etc/
cp -p /etc/smartd.conf $config_bk/etc/
cp -p /etc/samba/smb.conf $config_bk/etc/samba/
cp -p /etc/netplan/*.yaml $config_bk/etc/netplan/

# media_server directory is where docker compose files , all container config, cache directory
# BLD_hostname is where all the tools , scripts , config backup kept
# shutdown the conatiners
tar cvf $BKDIR/`hostname`-bk-`date -I`.tar bin media_server BLD_*

You could located the meda_server directory ( docker compose files, all containers files ) in a separate SSD drive.
Then you may not even have to restore the container from backup.
 
  • Like
Reactions: cesmith9999

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Make a shell script , run the shutdown-container and shell script using cron , run it nightly

# Config files backup directory
config_bk=$HOME/BLD_`hostname`/config_bk
BKDIR=backupfile-director

cd $HOME

mkdir -p $config_bk/etc/
mkdir -p $config_bk/etc/samba
mkdir -p $config_bk/etc/netplan

cp -p /etc/hosts $config_bk/etc/
cp -p /etc/exports $config_bk/etc/
cp -p /etc/environment $config_bk/etc/
cp -p /etc/fstab* $config_bk/etc/
cp -p /etc/smartd.conf $config_bk/etc/
cp -p /etc/samba/smb.conf $config_bk/etc/samba/
cp -p /etc/netplan/*.yaml $config_bk/etc/netplan/

# media_server directory is where docker compose files , all container config, cache directory
# BLD_hostname is where all the tools , scripts , config backup kept
# shutdown the conatiners
tar cvf $BKDIR/`hostname`-bk-`date -I`.tar bin media_server BLD_*

You could located the meda_server directory ( docker compose files, all containers files ) in a separate SSD drive.
Then you may not even have to restore the container from backup.
Thanks for this, I do plan to run all my docker related data (compose files, containers, appdata) on my optane drives which will be separate from the rootfs drive.

However, I was more interested in your ansible playbook related to the initial setup of the OS and packages :D.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
+1. The bug that causes pvestatd to wake up the drives is actually a bug in LVM and excluding them does indeed allow them to spin down.
Do you know if this affects remotely mounted shares to proxmox as well? I have SMB shares mounted to Proxmox and I'm seeing the drives never spin down on that remote server.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Do you know if this affects remotely mounted shares to proxmox as well? I have SMB shares mounted to Proxmox and I'm seeing the drives never spin down on that remote server.
Thats one I can't answer. I normally host shares on my proxmox server and don't have any remote mounted to it (and if I did they would likely be nfs or cephfs rather than SMB).

Sorry I can't be helpful on this one.
 

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
@IamSpartacus what was the final software platform you went with? Still Ubuntu host with Proxmox for VM as in the first post, or did you go in a different direction? Did you get spin up/down sorted out?

I'm working on a build for tinkering with Docker and VMs that I can't seem to get Unraid working on, but Ubuntu runs on just fine. I knew it would be idle much of the time, so had thought Unraid for ease of deployment and drive spin-down. But since Unraid isn't cooperating and Ubuntu is, I'm leaning toward MergerFS+Snapraid so I can still (hopefully) get spindown. But, its been a while since I used either of them, so I'm unsure how I want to proceed. Any advice you'd be willing to share would be appreciated!
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
@IamSpartacus what was the final software platform you went with? Still Ubuntu host with Proxmox for VM as in the first post, or did you go in a different direction? Did you get spin up/down sorted out?

I'm working on a build for tinkering with Docker and VMs that I can't seem to get Unraid working on, but Ubuntu runs on just fine. I knew it would be idle much of the time, so had thought Unraid for ease of deployment and drive spin-down. But since Unraid isn't cooperating and Ubuntu is, I'm leaning toward MergerFS+Snapraid so I can still (hopefully) get spindown. But, its been a while since I used either of them, so I'm unsure how I want to proceed. Any advice you'd be willing to share would be appreciated!
Im still in the testing phases as I had a second child a few months ago and have had no time to really move forward with my plans. Right now I'm testing a proxmox base os install with docker, mergerfs and snapraid. I ended up not liking all the changes that have been in the latest versions of Ubuntu. Debian is more comfortable to me. However I've hit a snag with getting nvidia drivers to load in Proxmox.
 
  • Like
Reactions: Markess

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
@Markess I've made some really good progress over the past few days. Since Monday I've done the following in proxmox:
  1. Got nvidia drivers installed and working for GPU transcoding in plex/emby/tdarr
  2. Installed and configured mergerfs. Currently running a mergerfs.balance to balance out the data on my disks since I added 3 new 10TB disks
  3. Installed and configured snapraid. Waiting for the mergerfs.balance to complete before running my first sync
  4. Installed and configured hd-idle for drive spin-down. hdparm does not work for my drives (WD Golds and WD Whites)
Once the snapraid parity sync is complete I'll be migrating all my "production" containers (swag, organizr, plex, tautulli, etc.) from my backup server over to this EPYC Proxmox server.
 
  • Like
Reactions: Markess

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
@Markess I've made some really good progress over the past few days. Since Monday I've done the following in proxmox:
  1. Got nvidia drivers installed and working for GPU transcoding in plex/emby/tdarr
  2. Installed and configured mergerfs. Currently running a mergerfs.balance to balance out the data on my disks since I added 3 new 10TB disks
  3. Installed and configured snapraid. Waiting for the mergerfs.balance to complete before running my first sync
  4. Installed and configured hd-idle for drive spin-down. hdparm does not work for my drives (WD Golds and WD Whites)
Once the snapraid parity sync is complete I'll be migrating all my "production" containers (swag, organizr, plex, tautulli, etc.) from my backup server over to this EPYC Proxmox server.

Thanks for the update! You've definitely saved me some time and frustration around hdparm. Half my disks are HD Whites. Also, I'd not worked with mergerfs-tools before, so I'll be checking out margerfs.balance and the others.

EPYC, I'm jealous! Keep posting on this so I can live vicariously through you :) . The board I'd been using for this project is just too flaky to trust with data, so I'm having to dig through the "parts closet" something else to use. The wife is working from home and the kids are virtual schooling now, so its not like I can easily sneak any more deliveries in unnoticed :p Leaving out the dual CPU boards (I was hoping for compact and relatively quiet), I've got it down to a Sandy Bridge board for mobile CPUs (QM67) or a PIledriver generation Opteron (AM3+). The Intel board is still "new" at least, but both are a step backward...so not very "epic".

Cheers!
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
A sub $50 G5400 CPU transcode Emby streams nicely
Yup, iGPU all the way if you're separating out those duties (which I was previously doing on an Intel Xeon E-2246G). But if you are doing a single AIO server, iGPU is not an option since you can't get more than 8c/16t and any more than 16 PCIe lanes.
 
  • Like
Reactions: Markess

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
Do what I do, I show my wife the regular price from Amzon $251, but only paid $53.
Oh, I've tried stuff like that before, but she's got a financial background & also knows I almost never buy this stuff retail. Plus, my kids are in their teenage sarcastic phase: when I recently upgraded my desktop, my daughter "helpfully" held up her phone's calculator with a 0 on it, and said, "Look, here's how much you will pay if you don't buy it at all." Coal in the stockings all around this year! :p

Realistically, I can probably buy what I want. But, I thought I'd save the next buying spree for something more exciting than another backup & tinkering/experimentation. I might just give in and go with the SM X9D or ASUS Z9 I'm not using. It won't be compact, energy efficient or quiet, but a pair of 8C CPUs, 128GB of RAM, and a couple old workstation GPUs will be a lot more flexible than the limited weak-sauce options I was considering. I've got an ancient 3U chassis and a spare 80+ Platinum ATX supply in the parts closet too...maybe there's a fun project there, trying to cram that in the 3U chassis? ;)
 
  • Like
Reactions: dawsonkm and Marsh

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
@Markess I've made some really good progress over the past few days. Since Monday I've done the following in proxmox:
  1. Got nvidia drivers installed and working for GPU transcoding in plex/emby/tdarr
  2. Installed and configured mergerfs. Currently running a mergerfs.balance to balance out the data on my disks since I added 3 new 10TB disks
  3. Installed and configured snapraid. Waiting for the mergerfs.balance to complete before running my first sync
  4. Installed and configured hd-idle for drive spin-down. hdparm does not work for my drives (WD Golds and WD Whites)
Once the snapraid parity sync is complete I'll be migrating all my "production" containers (swag, organizr, plex, tautulli, etc.) from my backup server over to this EPYC Proxmox server.
I'm going to try this over the weekend! Just so I understand your approach (because you have a MUCH better handle on this than I do), the Nvidia driver installation and disk configuration is done on the host (Proxmox)? Thanks!
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
I'm going to try this over the weekend! Just so I understand your approach (because you have a MUCH better handle on this than I do), the Nvidia driver installation and disk configuration is done on the host (Proxmox)? Thanks!
Everything is done on the host yes.
 
  • Like
Reactions: Markess

Jiovani

New Member
Jun 10, 2018
4
0
1
@IamSpartacus - Shall we be expecting a new thread with your current incarnation of your latest server setup?? Maybe with some rack pics? :) I am sure you are busy, but always enjoy hearing about your most current setup details. Have you consolidated back down to an AIO w/ your Epyc 7452 system?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
@IamSpartacus - Shall we be expecting a new thread with your current incarnation of your latest server setup?? Maybe with some rack pics? :) I am sure you are busy, but always enjoy hearing about your most current setup details. Have you consolidated back down to an AIO w/ your Epyc 7452 system?
I'm not sure it warrants a new thread but I will post an update to what I'm currently running in my previous AIO EPYC 7452 thread this week.
 
  • Like
Reactions: abq