Help Needed Deciding On Software Platform

[t0ny84]

Hi All,

I have an HP ML350e Gen8 V2 Server with a Dell H200e SAS HBA and NetApp Storage Array connected with 12 x 4TB HDDs.

Until I can find reasonably priced additional drive cages for the ML350e or other server hardware I will be using the NetApp Storage Array to hold the drives.

I currently have a simple Proxmox server setup and like the ability to create (and destroy) containers and VMs whilst I am learning without impacting other running systems. I want to run (at a minimum) NGINX (or similar), Sonarr, Radarr and Prowlarr along with the basic NAS \ File Storage. Apart from streaming the videos from the system, it will be mainly for storage.

In researching options I have narrowed down my choices to the below 3 options, each has its pros and cons. I was hoping to get some insight from more experienced forum members who might be able to offer opinions \ thoughts as to which would be the better option.

Option 1 - TrueNAS Scale with RAID 5 or 6
Pros
* Ease of setup and use.
* Single or Double Parity protection.

Cons
* Drive sizes need to stay the same.
* Cannot add "random" hard drives.
* Only able to use apps built into TrueNAS.

Option 2 - Ubuntu Server + JBOD + SnapRaid + Symlink Script or MergeFS
Ubuntu Server and connect to the Storage Array as a JBOD.
Use SnapRaid to provide "raid parity protection"
Either use a script to Symlink to create a "shortcut" folder of all the folders on the drives so that anything I connect to it will see only 1 folder or use MergeFS.

Pros
* Drive sizes can vary.
* Can add additional hard drives irrespective of their size.

Cons
* Increased difficulty in setting up.
* Possibility to lose files if SnapRaid hasn't been run.
* Increased degree of difficulty in setting up.
* Increased probability of issues arising.

Option 3 - Proxmox with Ubuntu Server or TrueNAS Server
Setup Proxmox and create an Ubuntu Server or TrueNAS Server then pass through the HBA \ Hard Drives to this system to control.

Pros
* Same Pros as Option 2
* Allows even more flexibility.
* Can spin up additional VMs\Containers and not interfere with other containers \ VMs.

Cons
* Same Cons as Option 2
* The highest degree of difficulty in setting up.
* The highest probability of issues arising.

t0ny84

 

[t0ny84]

Also if I could also ask:

Is RAID6 over RAID5 worth the additional space requirements?

If I were to set up Ubuntu Server \ TrueNAS on Proxmox if my boot drive died can I just reinstall either TrueNAS on Proxmox or TrueNAS directly and then rebuild \ access my RAID drives?

EXT4 or ZFS, benefits of one over the other?

 

[i386]

Raid is for uptime/availability, backups for keeping data safe.
Raid 5/Z1 can handle 1 failed device and still calculate the missing data, raid 6/z2 can handle 2 failed devices and z3 can handle 3 failed devices.
(The question of raid 5 vs 6 is kinda moot because you have backups and can always restore the data...)

The question of zfs or anything else will always result in the same/similar posts and how zfs is a magical silver bullet solving all problems
You have to know what you need, the trade offs are and if you can live with that. Zfs makes a lot of things easier and a lot of things more difficult in my opinion.
One point for me to not choose zfs was the point you mentioned about expandibility and how you can not add a single device to an existing vdev. This was solved in a "recent" update to zfs, but it doesn't recalculate the existing data and won't give you the storage efficiency of the new stripe widths for old data.

 

[gea]

Questions like raid level redundancy, raid type or ECC RAM affects the question if you want to accept or minimize a certain risk for your data.How valuable are your data and what is your overall backup strategy. In the end there are the two main options:

1.) Data are not too valuable and for some important ones, there are external backups.

Remains the risk that in case you need them, they are not current/ too old or bad. Can happen when transfers or data are not checksum protected.If In case of a 40TB fileserver with 12 disks, a bad strategy especially as the disks are propably quite old, there is hardy a 40BT backup nd even when a restore will last many days.

2.) Data is valuable in general

and it is often hard to decide which ones in the 40TB pool are valuable so you should use any affordable option to avoid a pool failure with a dataloss. No need for restore from backup ist the best strategy paired with enough snaps to have access to data versions for certain times in the past. Under such a strategy, raid is not for uptime/availability but the base to protect data.

With 12 old disks it is absolutely out of question for me to use raid-6 or Z2 (Truenas is ZFS and there is no raid 5/6 option but Z1/2/3). Sun decided to name them different to make it clear that the main risk of raid 5/6, partially written raid stripes only on some disks in case of a crash are avoided with ZFS and Copy on Write.

With 12 disks your can create a single Z2 (40TB capacity) or two Z2 with 6 disks (twice the performance, 32 TB).
Despite the very low propability of a dataloss in a Z2, it does not protect against a disaster like fire, theft or amok hardware. Even with ZFS Z2 you need a disaster backup for important data, minimum a removable USB disk with a ZFS backup pool of say 10TB

 

[t0ny84]

Thank you both for your input, my understanding with RAID is you could get a level of redundancy but have all the drives work as 1 big drive/pool rather than in this case 12 smaller drives.

RAID5 = RAID-Z
RAID6 = RAID-Z2

Whilst I would love to have larger drives or SSDs I have to work with what I have.
My thoughts are now moving away from TrueNAS and to possibly OMV with EXT4 or BTRFS.


t0ny84

 

[unwind-protect]

RAID6 is absolutely worth it over RAiD5. The problem is that after a drive failure you rsync a new drive. And the resync puts a lot of stress on the remaining old drives. At a time where one more drive failure takes your array offline.

As for OS, I would use ZFS on FreeBSD, possibly inside Proxmox.

 

[louie1961]

Why are those your only three options? How good are your Linux skills? I mean you could do whatever you want with a plain vanilla install of Ubuntu, Debian, RHEL, etc. Install Docker and Portainer to run whatever containers you want. Create RAID arrays with ZFS (although with ZFS, you need to use something like the Proxmox kernel or recompile your own), or MDADM/LVM. and then just share out either an NFS share or a CIFS share for storage. If you want to use BTRFS, I would advise going the way that Synology has gone to implement BTRFS: create the raid with MDADM, use LVM to chop up the array into smaller chucks, and then use BTRFS only for the file system and snapshots. Lots of different options if you are skilled enough

One thing I wouldn't do is run TrueNAS on top of Proxmox. I know people do it, but why not just run TrueNAS on bare metal? Saves you the hassle of doing hardware pass through.

 

[gea]

As a "Linux" TrueNAS is very inflexible as a lot of features are locked inside the web-gui. Avoid settings from console as I have seen when I tried TrueNAS as a member of my ZFS Web-GUI with User, ZFS or Share settings. Proxmox on the other is ultra flexible with a very good VM handling and not locked into "VM server only". If you do not need feautures from the TrueNAS Gui you can use pure Proxmox instead, optionally with an additional Web-Gui for ZFS management as the Proxmox Web-Gui is focussed on VM management.

btw.
ZFS is superiour to mdadm/lvm regarding raid security and to btrfs regarding stability and features.

 

[louie1961]

btw.
ZFS is superiour to mdadm/lvm regarding raid security and to btrfs regarding stability and features.

Well, that is one person's opinion, that is not shared by Synology (with a market leading 27% market share for NAS) or their legions of fans.

 

[gea]

Market Share is related to marketing not technical state of the art features.

This is why they do not adopt the two main advantages of modern filesystems like self healing filesystems due checksum checks on data errors (They can't beside Metadata unless btrfs raid is stable on higher raid levels) and Copy on Write to avoid incomplete Raid stripes for same reason.

That they do not offer ECC on most smaller NAS models despite beeing quite expensive is another point as well as superiour Hybrid pools with special vdevs or safe software based sync write or draid a raid level for 100+ disks is another plus for ZFS.

Btrfs is good but not as good as ZFS and even the features it has are not implemented in the really relevant points.

 

[unwind-protect]

Well, that is one person's opinion, that is not shared by Synology (with a market leading 27% market share for NAS) or their legions of fans.

mdadm is very good software but since it separates filesystem and raid it has the "raid hole". Not their fault but a disadvantage nontheless.

Never ran btrfs and probably never will.