Marriage = Storage Consolidation

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
I do at home. My home Proxmox node is just for goofing around with :) I have a 3 node Proxmox cluster colocated at a local datacenter that I use Ceph on for shared storage. I host all my production VMs there.
I see. Yea my home cluster IS my "production" cluster ;). But as much as I like to plan for max uptime, I'm starting to realize there's only so far I'm willing to go for my home network. And having a shared bulk storage server and a separate shared VM storage server is just overkill. I want to either combine them into one physical box or move my VM storage back locally onto the hosts. I'm honestly learning towards #2 for simplicity sake since I just don't have the time I used to to spend on my "toys" (see thread title :D). Since I use Veeam I can bring a VM back up in a matter of minutes on a different host if one host was to become unavailable for whatever reason and that should be good enough for at home.
 

Stux

Member
May 29, 2017
30
10
8
46
Why not just get rid of the unraid and put the bulk and vm store under FreeNAS?

You can use different pools.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Why not just get rid of the unraid and put the bulk and vm store under FreeNAS?

You can use different pools.
I've considered it but based on my use case (mainly media serving), non-striped pooling of my data seems to fit best. Not only will performance be better (streaming multiples files stored on different disks as opposed to lots of random IO from a multi-disk striped array, but I also have the added security of not losing all my data (we're talking 60TB and growing) if I lose more disks than I have parity.
 

CookiesLikeWhoa

Active Member
Sep 7, 2016
112
26
28
35
I've considered it but based on my use case (mainly media serving), non-striped pooling of my data seems to fit best. Not only will performance be better (streaming multiples files stored on different disks as opposed to lots of random IO from a multi-disk striped array, but I also have the added security of not losing all my data (we're talking 60TB and growing) if I lose more disks than I have parity.
I don't see how a non-striped pool would perform better over a striped pool, since a non-striped pool is limited to IOPs/throughput of 1 disk, but I digress.

I would vote for some flavor of ZFS, either FreeNAS or ZFS on Linux. With the new feature coming to openZFS where you can expand a pool and have it rebalance I can't think of any reason to not run ZFS. (OpenZFS on Twitter)
 

nitrobass24

Moderator
Dec 26, 2010
1,087
131
63
TX
I don't see how a non-striped pool would perform better over a striped pool, since a non-striped pool is limited to IOPs/throughput of 1 disk, but I digress.

I would vote for some flavor of ZFS, either FreeNAS or ZFS on Linux. With the new feature coming to openZFS where you can expand a pool and have it rebalance I can't think of any reason to not run ZFS. (OpenZFS on Twitter)
It’s all about your use-case. For him streaming a media file is sequential IO. So on a single disk this is no problem. Streaming a second file, if it’s on the same disk the read pattern becomes random as it’s trying to read multiple files. However in a non-striped Raid you have a chance the second stream is located on a second disk and you still have two sequential operations.

In a striped scenario you are almost always going to be doing random reads.

For media playback ZFS doesn’t buy you anything since ARC and even L2ARC are not read-ahead caching, so unless you are playing your file a second time it will always be pulled from disk. So you still end up being limited to single drive performance (assuming single vdev).


Sent from my iPhone using Tapatalk
 

CookiesLikeWhoa

Active Member
Sep 7, 2016
112
26
28
35
It’s all about your use-case. For him streaming a media file is sequential IO. So on a single disk this is no problem. Streaming a second file, if it’s on the same disk the read pattern becomes random as it’s trying to read multiple files. However in a non-striped Raid you have a chance the second stream is located on a second disk and you still have two sequential operations.

In a striped scenario you are almost always going to be doing random reads.
I under stand that. The question then is how many streams does he have at once? If it's regularly more than 1, then it would seem that a striped array would be the clear winner. Even then, I really question if a single disk's sequential IO is greater than an arrays random IO. Though I guess it depends on the number of disks and why type of array we are talking about. (A RaidZ2 array with 2 vdevs, each consisting of 5 drives, would likely have greater random read IO than a single drive's sequential IO for media, I would think)

For media playback ZFS doesn’t buy you anything since ARC and even L2ARC are not read-ahead caching, so unless you are playing your file a second time it will always be pulled from disk. So you still end up being limited to single drive performance (assuming single vdev).
Again, I don't think performance is the right question here. I agree ARC/L2ARC would be useless for a media server, but that's not really the point of using a ZFS based file system for media. It's more about data integrity.

If there is no point in data integrity then I can see why you would want to use JBOD. It would give the most raw storage; but I seem to recall the OP talking about parity earlier in the thread.
 
  • Like
Reactions: Stux

bitrot

Member
Aug 7, 2017
95
25
8
One negative aspect of striped arrays when it comes to home media servers is power consumption, an often overlooked factor. In a non–striped array, only the disks containing the media that is accessed needs to spin, not all of them as is the case in a striped array. Depending on the number of disks (and the kind of disks you’re using) in your array, the power saved can be quite significant.

Another advantage of non–striped arrays, particularly in comparison to ZFS pools, is the relative ease and cost effectiveness of expanding storage.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
..

For media playback ZFS doesn’t buy you anything since ARC and even L2ARC are not read-ahead caching, so unless you are playing your file a second time it will always be pulled from disk. So you still end up being limited to single drive performance (assuming single vdev).
For L2Arc you can enable read ahead with a setting in /etc/system (Solarish)
Code:
set zfs:l2arc_noprefetch=0
This is why L2Arc can be an advantage even with a lot of RAM
 

Joel

Active Member
Jan 30, 2015
850
191
43
42
I under stand that. The question then is how many streams does he have at once? If it's regularly more than 1, then it would seem that a striped array would be the clear winner. Even then, I really question if a single disk's sequential IO is greater than an arrays random IO. Though I guess it depends on the number of disks and why type of array we are talking about. (A RaidZ2 array with 2 vdevs, each consisting of 5 drives, would likely have greater random read IO than a single drive's sequential IO for media, I would think)
RaidZ2 has MUCH better READ performance than a single drive, its WRITE performance is limited to the number of vdevs.

One negative aspect of striped arrays when it comes to home media servers is power consumption, an often overlooked factor. In a non–striped array, only the disks containing the media that is accessed needs to spin, not all of them as is the case in a striped array. Depending on the number of disks (and the kind of disks you’re using) in your array, the power saved can be quite significant.

Another advantage of non–striped arrays, particularly in comparison to ZFS pools, is the relative ease and cost effectiveness of expanding storage.
True statements all. ZFS pools have the other advantage of bitrot protection though. Not a huge factor for a video library perhaps, but for my DSLR photos it is something very attractive.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
RaidZ2 has MUCH better READ performance than a single drive, its WRITE performance is limited to the number of vdevs.
If I'm readying 6 files off a single 8 disk RAIDz2 array, vs reading 6 files off 6 different drives in non-striped array, you're saying the RAIDz2 performance would be much better?
 

markarr

Active Member
Oct 31, 2013
421
122
43
True statements all. ZFS pools have the other advantage of bitrot protection though. Not a huge factor for a video library perhaps, but for my DSLR photos it is something very attractive.
I know snapRAID does and I think a couple others have bitrot protection. Its also significantly easier to expand non-striped, as all you do is add the drive to the config and run a sync and its added, you can add disks one at a time to as many as you want.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
ZFS proponents often tout bitrot as a huge feature. While I definitely see it as an important consideration for enterprise/production servers and irreplaceable data, it's honestly not a factor at all for me when talking about a media server.

For me VMs, ZFS is my choice. For my 50TB of media, not at all necessary. But then again I have gigabit internet so the thought of having to "replace" my media isn't so daunting :cool:.
 

Joel

Active Member
Jan 30, 2015
850
191
43
42
If I'm readying 6 files off a single 8 disk RAIDz2 array, vs reading 6 files off 6 different drives in non-striped array, you're saying the RAIDz2 performance would be much better?
In this scenario, I am assuming you're talking about a video server playing movies?

Hmm, in this hypothetical, maybe not, because now we're comparing sequential (NSA) vs. random (ZFS). I suspect it would be fairly close though. In the real world though, ZFS would still be more flexible.
- Can you guarantee that all six files that you want to access will be on different drives?
- What if three files are on a single drive?

So in the end, you're asking what's best, and the answer is always "It depends..." and it's really something that only you can decide.
 

Joel

Active Member
Jan 30, 2015
850
191
43
42
For me VMs, ZFS is my choice. For my 50TB of media, not at all necessary. But then again I have gigabit internet so the thought of having to "replace" my media isn't so daunting :cool:.
Agree, which is exactly why I said...

Not a huge factor for a video library perhaps, but for my DSLR photos it is something very attractive.