Adding Storage to Server

[IronedSoul]

So I bought this server off ebay about a week ago for a budget project for some of my storage. Im currently using 66TB between dropbox and onedrive. Im trying to move my storage off the cloud and onto this server which is hosted in my RV.

The problem is I don't have the budget to purchase a shit load of hard drives at once I can probably buy 4 x 10tb of so. My question is would it be a way for me to transfer some data at a time and as well add more storage to the RAID arrays as im able to purchase more storage drives without losing any transfered data?

Im not a specialist in any of this but I'm for sure trying to save some money on paying out the ass for dropbox and onedrive. I've installed promox on the server so I can delegate where the files and such are going and I'm using youtube for that. I've been doing some google research on the raid arrays and stuff like that as well but I cant seem to find and firm answers or any software that helps me out.

Any help would be greatly appreciated.

 

[nexox]

The details depend on exactly what RAID system you're using (it's going to be software RAID with that LSI card, this is generally a good thing,) but it's pretty standard to accommodate new disks to expand an array. You will likely want to select the RAID system based on other criteria and then just look up how to grow an array in that system once you find one you like.

 

[IronedSoul]

The details depend on exactly what RAID system you're using (it's going to be software RAID with that LSI card, this is generally a good thing,) but it's pretty standard to accommodate new disks to expand an array. You will likely want to select the RAID system based on other criteria and then just look up how to grow an array in that system once you find one you like.

I plan to use 8 x 10TB SAS 3 HUH721010AL4200 and 2 x ST10000NM0126 Seagate 10TB. So it totall im looking for 100TB of space.

Im still looking for which raid array to use I need at least 80TB usable and 20TB or so for backup incase a drive goes down of some sorts.

Any recommendation on what raid to use for that type of setup?

 

[pricklypunter]

There are many flavours of storage schemes out there, including RAID, all have their upsides and downsides. No matter what you use, you will have to take into account the formatted capacity of your devices, plus whatever overhead your chosen schema will require. For example 8 x 10TB disks will not yield 80TB usable, being that each disk has about 9.1TB (roughly) formatted capacity. Add in any redundancy overhead for your chosen array type and you can see where we are heading...

Given disk capacity nowadays, RAID5 and similar parity based schemes carry higher risks of data loss, in the event that it all goes wrong. While RAID5 schemes do still have a place for some use cases, I would not recommend using it. RAID6 should be considered a starting point now, for any sort of parity based RAID. If performance is your goal, rather than outright capacity, you are probably wanting to use a disk mirror scheme. As you mentioned that you are playing in the Proxmox sand pit, that comes with ZFS rolled right in for storage, so you could use RAIDZ6 for example, to build your arrays, but there are plenty of other storage schemes out there to choose from, only you can decide how to proceed

 

[i386]

Raid is not a backup. It improves availability/uptime but it doesn't prevent data loss.


Zfs has recently added the functionality to expand a raidz volume, I would wait a few months before trying it. Older zfs versions can only add devices by adding them in groups (vdev). A hack to add more storage was to replace existing devices with larger ones, one by one.

mdadm (linux software raid) can expand and shrink a volume

(windows software raid sucks and storage spaces has performance problems with default settings and party volumes)

 

[louie1961]

How often does your data change and to what magnitude? Assuming that you are not storing 66TB of new data each month and most of it stays static, I would NOT use RAID in an RV setup. RAID really serves one purpose: to provide uptime reliability. That is making sure your server stays up or your data stays available. As noted, RAID is not backup. If I were you I would go without the RAID and focus on backing up your 66 TB to cheaper places in the cloud, using cheaper archival storage for your backup copies. For instance, Amazon S3 Glacier Deep Archive costs $0.00099 per GB per month. Yes it takes 12 hours to restore your data, and yes there are other costs associated for data transfers and the like. But they provide 99.99999% durability of your data by storing multiple copies in multiple availability zones. The chance of losing data is like 1 object lost every 10,000 years or something like that. Azure, GCP, and others provide similar archival storage. I would follow the 3-2-1 back up strategy: 3 copies, 2 onsite, one offsite, and I would use Glacier for the offsite.

 

[IronedSoul]

There are many flavours of storage schemes out there, including RAID, all have their upsides and downsides. No matter what you use, you will have to take into account the formatted capacity of your devices, plus whatever overhead your chosen schema will require. For example 8 x 10TB disks will not yield 80TB usable, being that each disk has about 9.1TB (roughly) formatted capacity. Add in any redundancy overhead for your chosen array type and you can see where we are heading...

Given disk capacity nowadays, RAID5 and similar parity based schemes carry higher risks of data loss, in the event that it all goes wrong. While RAID5 schemes do still have a place for some use cases, I would not recommend using it. RAID6 should be considered a starting point now, for any sort of parity based RAID. If performance is your goal, rather than outright capacity, you are probably wanting to use a disk mirror scheme. As you mentioned that you are playing in the Proxmox sand pit, that comes with ZFS rolled right in for storage, so you could use RAIDZ6 for example, to build your arrays, but there are plenty of other storage schemes out there to choose from, only you can decide how to proceed

Thanks for the insight I for sure want to maxamize the space. I do understand I won't get a full 100TB out of 10x10 TB's. I'm looking into some redundancy options right now but that's for later down the line considering I still have a couple month left on both of my current subscriptions that's going to give me at least a couple months to pan out this project. Ill try out raid 6 though from what I've googled it seems like a good starting point like you said.

How often does your data change and to what magnitude? Assuming that you are not storing 66TB of new data each month and most of it stays static, I would NOT use RAID in an RV setup. RAID really serves one purpose: to provide uptime reliability. That is making sure your server stays up or your data stays available. As noted, RAID is not backup. If I were you I would go without the RAID and focus on backing up your 66 TB to cheaper places in the cloud, using cheaper archival storage for your backup copies. For instance, Amazon S3 Glacier Deep Archive costs $0.00099 per GB per month. Yes it takes 12 hours to restore your data, and yes there are other costs associated for data transfers and the like. But they provide 99.99999% durability of your data by storing multiple copies in multiple availability zones. The chance of losing data is like 1 object lost every 10,000 years or something like that. Azure, GCP, and others provide similar archival storage. I would follow the 3-2-1 back up strategy: 3 copies, 2 onsite, one offsite, and I would use Glacier for the offsite.

Hey thanks for the feedback its not new data everymonth its the same data pretty much I add maybe 1TB every 6 months or so but it is data that I don't wanna lose. The main thing I want it to be able to access the files via GUI and not lose any and the ability to add more and more storage whenever I need it if that means consistently adding disk or adding a new server with more disk. Another of the main focuses is not having to pay a MRC for any services including backup and my RV runs partially solar so ill be saving some electric cost.

I don't need access to it all the time ill more than likely turn the server off when I don't need to access the storage, fix files, move files or anything like that so its really just sitting storage.

 

[cbarbie]

I have various disk configurations across multiple platforms (WinSvr, Unraid. Hardware Raid, Software Raid, StorageSpaces, Proxmox ZFS, etc). My simplest was Windows 10 with Stablebit Drivepool, Scanner, CloudDrive.

With DrivePool, per folder, you specify how many copies of the data you want and it will automatically maintain that many copies across your disks.
With Stablebit CloudDrive, one or more of the those copies can be stored in one or more of the cloud providers.
You can use have a mix of any size disks. This worked great when I had to wait for BF deals.
You can tell Drivepool to logically empty a disk and then you can replace it with a different disk.
If your hardware fails, you can move a drive to another machine and still access the data stored on that drive.

Stablebit does not have bitrot protection.