FreeNAS to AWS Glacier strategy

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Alfa147x

Active Member
Feb 7, 2014
189
39
28
Hey folks,

I'm looking to backup my FreeNAS store that houses my photo library backups and video files to Glacier. The issue is my connection is 60MB/s down but only 6MB/s up so i'm looking to stage backups to a 5 TB external USB 3.0 drive and then move them to Glacier for off-site backups. That way if all of my drives fail I have a nearline backup and then if server rack gets stolen I have slightly stale backups on Glacier.

I'm not a pro photographer and only have new media a few times a month which is why I'm hoping to use Glacier's cheaper storage. Any suggestions on how to conduct all of this? I could spin up another BSD Jail or is there a tool that could help?

Also if I use FreeNAS to create snapshots, will that provide me with a simple file to push to Glacier?

Thanks!
 

Biren78

Active Member
Jan 16, 2013
550
94
28
How much data do you actually have?

At 5-10TB you could almost pick up a cheap server, put FreeNAS on it and just do simple ZFS send/ receive to make it really easy on yourself.

Glacier retrieval times suck.
Q: How long does it take for jobs to complete?

Most jobs will take between 3 to 5 hours to complete.
 

Blinky 42

Active Member
Aug 6, 2015
615
232
43
48
PA, USA
The other factor is how much content do you add/change per day/week/whatever.

For 5-10TB it may be cheaper and easier to just have a pool of USB drives and rotate them through your family/friends/office for offsite backup.
The AWS calc shows ~$35/mo for 5TB of storage. You could buy 3x 5TB USB drives and rotate them for the same price over a year and make a complete backup in under a day whenever you want without waiting for the upload.

That said, googling for rsync glacier gives a number of starting points, like DanielKinsman/lincremental · GitHub
Choosing between them depends on how you want to handle the glacier side. Do you want to pull a full snapshot of everything at a point in time, or pull files one at a time.
 

Alfa147x

Active Member
Feb 7, 2014
189
39
28
The other factor is how much content do you add/change per day/week/whatever.

For 5-10TB it may be cheaper and easier to just have a pool of USB drives and rotate them through your family/friends/office for offsite backup.
The AWS calc shows ~$35/mo for 5TB of storage. You could buy 3x 5TB USB drives and rotate them for the same price over a year and make a complete backup in under a day whenever you want without waiting for the upload.

That said, googling for rsync glacier gives a number of starting points, like DanielKinsman/lincremental · GitHub
Choosing between them depends on how you want to handle the glacier side. Do you want to pull a full snapshot of everything at a point in time, or pull files one at a time.
Thanks for that rsync link it gives me a starting point. The data set currently is only about 500 GB but I'm expecting some growth. Fortunately I receive some AWS credits on a recurring basis so AWS costs are a lesser concern.
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
This doesn't really address the original question at all - but have you considered BackBlaze B2. It's more like standard AWS S3 storage (no multi-hour retreival penalty), but priced (I think) even lower than Glacier.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
I got an account at rsync.net (Denver location) and they upgraded me to their ZFS offering. One thing I am really liking is the ability to just use standard ZFS commands, scp and rsync to move files around. Fairly fast too.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
I'm starting to evaluate all these services as well, these ingest/egress/api fees can add up and are hard to calculate out total costs. B2 is nice and I am starting to test that more. Google storage was a good price option as well, and Dreamhost has their object storage too. B2 is cheapest now I think.
 
  • Like
Reactions: wsuff

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
I'm starting to evaluate all these services as well, these ingest/egress/api fees can add up and are hard to calculate out total costs. B2 is nice and I am starting to test that more. Google storage was a good price option as well, and Dreamhost has their object storage too. B2 is cheapest now I think.
Yea the hard part is figuring bandwidth out charges. 5TB/ mo data transfer out is not cheap on Amazon. Well, at least if you compare it to normal datacenter bandwidth costs.
 

Jon Massey

Active Member
Nov 11, 2015
339
82
28
37
This doesn't really address the original question at all - but have you considered BackBlaze B2. It's more like standard AWS S3 storage (no multi-hour retreival penalty), but priced (I think) even lower than Glacier.
B2 looks really nice, but still very much in private beta :(
 

Alfa147x

Active Member
Feb 7, 2014
189
39
28
A bit more context: I currently have a 2 drives in circulation between my parents and a friend but I'm looking to move across the country where I'll have no contacts for a little while. Also with the move I'm trying to cut expenses down and use resources that are freely available (aws credits!). The FreeNAS box consolidated 2x ReadyNAS appliances and my VMware server (Thank god for BSD Jails!)

From some attempts last night it appears that going to Glacier is going to be more difficult than I thought.

Now I'm considering S3 @ 1TB the prices are:
Standard Storage: $30
Standard - Infrequent Access Storage: $12.50
Reduced Redundancy Storage: $24
+ egress charges per 1TB download: $10​
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Some days I think I should just rent colo space for people to send their storage servers.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
If you're really looking for just backup, another option is any of the cloud-based backup services. That's what I do as one of my backup targets (along with a local 8TB archive drive to rsync to, etc..)

Your per-month cost can be as low as $5/month and they will do the deltas, etc for the backups. Many of them can seed the initial backup as well with a HDD mailed to them, saving time. I use Crashplan and their unlimited plan for this.
 
  • Like
Reactions: Patrick

Jon Massey

Active Member
Nov 11, 2015
339
82
28
37
...I use Crashplan and their unlimited plan for this.
What sorts of speeds do you get with Crashplan and from where?

We're using Backblaze at the moment for a project which is generating about 40GB of video per day per site which we need to archive until at least the end of next year but are currently being bitten by Backblaze's 30 day retention policy (i.e. when we delete from the acquisition boxes, it vanishes from BB 30 days later so we restore back down to a stack of 8TB archive drives at one site but that's pretty unsatisfactory). I see crashplan does infinite retention but I've heard mixed things about its speed - the good thing about backblaze (apart from the price) is that we can upload everything we need to in the hours between afternoon and morning milking. Our sites are all in the UK, one of which on a 50/50 leased line, two are connected to JANET - Wikipedia, the free encyclopedia and another in the the middle of BFN about to come online which we're probably going to have to sneakernet the data out of every week.