FreeNAS Cloud sync with delayed delete?

Discussion in 'FreeBSD and FreeNAS' started by SPCRich, Oct 9, 2019.

  1. SPCRich

    SPCRich Active Member

    Joined:
    Mar 16, 2017
    Messages:
    136
    Likes Received:
    34
    In FreeNAS (11.x U5, if that matters), I have a cloud sync task to upload a folder to GCP cold storage (my photography backup). I set the cloud sync task to "sync", so if I delete something locally, it gets deleted in the cloud. The problem is that GCP storage has a 90 day minimum storage period for cold storage, or else they charge you early delete fees (cost of storage per GB x remaining period of days to get to 90). I'm trying to figure out a way to do make it so that I don't get charged the early delete fees, and so far I only have one solution, wondering if there are any others.

    My solution: Schedule 2 cloud sync tasks:

    First: Run daily, with a "COPY" option, so that it just constantly uploads stuff to cold storage.
    Second: Run every 90 days ( 0 0 1 */3 *) that does a "SYNC". This would then prune out anything old, although if anything changed <90 days, it would still accumulate the early delete costs.

    Is there a better way of doing this? It may be cheaper to do nearline storage, which has a 30 day minimum period, but I'd still have the same issue there. Another option is instead of doing a daily upload, I do a weekly or monthly upload, since my images don't change that often (I take photos maybe once every 2-3 weeks), but then I'd need a local backup or secondary option just in case.
     
    #1
  2. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,843
    Likes Received:
    423
    Does a move to a deferent folder count as delete ? If not you could just move images to a trash folder and delete based on date, probably not what you want but just an idea.
     
    #2
  3. SPCRich

    SPCRich Active Member

    Joined:
    Mar 16, 2017
    Messages:
    136
    Likes Received:
    34
    I believe you'd get hit with 2 charges, early delete (of the delete from one folder) and a data retriveal fee (0.05$/GB) for a "READ, COPY, or REWRITE" of data or metadata.

    I'm going to run 2 tests for a month each, one to cold storage and one to nearline storage and see which is more cost effective.. based on the fact that nearline is 30days minimum vs 90 days, and "READ/COPY/RW" fees are 0.01$ vs 0.05$, it may actually be cheaper to store it nearline. both options at least on paper appear to be cheaper than backblaze, which currently is charging me around 50$/month for ~1.3TB of images.
     
    #3
  4. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,843
    Likes Received:
    423
    Your probably right about the costs. Did you price out own backup to S3 by chance ?

    Wow $50 for 1.3tb is a bit much.
    That’s dedicated server prices with a few TB of you own space.
     
    #4
  5. zer0sum

    zer0sum Active Member

    Joined:
    Mar 8, 2013
    Messages:
    258
    Likes Received:
    77
    Are you re-uploading all 1.3TB every month?
    Because if you're just syncing file changes and adding a moderate amount of new files, the pricing should be way cheaper.

    upload_2019-10-11_8-50-20.png
     
    #5
  6. Nugget

    Nugget New Member

    Joined:
    Jul 13, 2017
    Messages:
    17
    Likes Received:
    17
    Using Backblaze B2 can also solve the delayed deletion concern. You can apply complex lifecycle rules to your backup bucket to retain revision history on files and control how long stale/deleted files are retained before being removed from the sync.

    This will all work fine underneath the rclone sync that FreeNAS manages.

    One caveat is that there's a known limitation with FreeNAS's B2 support that can result in much higher than expected class c transaction costs if you're syncing tons of files (like a git repo hash or some other mechanically-generated data store). This will be fixed in 11.3 (see this issue). I run rclone "by hand" in a jail for now to get around the limitation. If you've got a normal directory of normal files it probably won't be an issue for you, but it's smart to keep a close watch on your class c transaction costs when you first set up the cloud sync just to be safe.
     
    #6
Similar Threads: FreeNAS Cloud
Forum Title Date
FreeBSD and FreeNAS FreeNAS Corral and NextCloud - Silly Easy to Install with Docker Mar 17, 2017
FreeBSD and FreeNAS Running ZFS on Ubuntu vs Freenas and ZFS in the cloud Nov 25, 2016
FreeBSD and FreeNAS Need some FreeNAS Owncloud plugin help Oct 20, 2015
FreeBSD and FreeNAS Windows, FreeNAS, eSATA, Port Multipliers, Non-IT RAID & VMs Sep 20, 2019
FreeBSD and FreeNAS Need FreeNAS sharing help Jul 4, 2019

Share This Page