FreeNAS Cloud sync with delayed delete?

SPCRich

Active Member
Mar 16, 2017
235
85
28
40
In FreeNAS (11.x U5, if that matters), I have a cloud sync task to upload a folder to GCP cold storage (my photography backup). I set the cloud sync task to "sync", so if I delete something locally, it gets deleted in the cloud. The problem is that GCP storage has a 90 day minimum storage period for cold storage, or else they charge you early delete fees (cost of storage per GB x remaining period of days to get to 90). I'm trying to figure out a way to do make it so that I don't get charged the early delete fees, and so far I only have one solution, wondering if there are any others.

My solution: Schedule 2 cloud sync tasks:

First: Run daily, with a "COPY" option, so that it just constantly uploads stuff to cold storage.
Second: Run every 90 days ( 0 0 1 */3 *) that does a "SYNC". This would then prune out anything old, although if anything changed <90 days, it would still accumulate the early delete costs.

Is there a better way of doing this? It may be cheaper to do nearline storage, which has a 30 day minimum period, but I'd still have the same issue there. Another option is instead of doing a daily upload, I do a weekly or monthly upload, since my images don't change that often (I take photos maybe once every 2-3 weeks), but then I'd need a local backup or secondary option just in case.
 

Evan

Well-Known Member
Jan 6, 2016
3,278
566
113
Does a move to a deferent folder count as delete ? If not you could just move images to a trash folder and delete based on date, probably not what you want but just an idea.
 

SPCRich

Active Member
Mar 16, 2017
235
85
28
40
Does a move to a deferent folder count as delete ? If not you could just move images to a trash folder and delete based on date, probably not what you want but just an idea.
I believe you'd get hit with 2 charges, early delete (of the delete from one folder) and a data retriveal fee (0.05$/GB) for a "READ, COPY, or REWRITE" of data or metadata.

I'm going to run 2 tests for a month each, one to cold storage and one to nearline storage and see which is more cost effective.. based on the fact that nearline is 30days minimum vs 90 days, and "READ/COPY/RW" fees are 0.01$ vs 0.05$, it may actually be cheaper to store it nearline. both options at least on paper appear to be cheaper than backblaze, which currently is charging me around 50$/month for ~1.3TB of images.
 

Evan

Well-Known Member
Jan 6, 2016
3,278
566
113
Your probably right about the costs. Did you price out own backup to S3 by chance ?

Wow $50 for 1.3tb is a bit much.
That’s dedicated server prices with a few TB of you own space.
 

zer0sum

Active Member
Mar 8, 2013
514
214
43
Are you re-uploading all 1.3TB every month?
Because if you're just syncing file changes and adding a moderate amount of new files, the pricing should be way cheaper.

upload_2019-10-11_8-50-20.png
 

Nugget

New Member
Jul 13, 2017
26
19
3
Houston, TX
keybase.io
Using Backblaze B2 can also solve the delayed deletion concern. You can apply complex lifecycle rules to your backup bucket to retain revision history on files and control how long stale/deleted files are retained before being removed from the sync.

This will all work fine underneath the rclone sync that FreeNAS manages.

One caveat is that there's a known limitation with FreeNAS's B2 support that can result in much higher than expected class c transaction costs if you're syncing tons of files (like a git repo hash or some other mechanically-generated data store). This will be fixed in 11.3 (see this issue). I run rclone "by hand" in a jail for now to get around the limitation. If you've got a normal directory of normal files it probably won't be an issue for you, but it's smart to keep a close watch on your class c transaction costs when you first set up the cloud sync just to be safe.