Storage in Datacenter and "Cache" at home

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

LL0rd

New Member
Feb 25, 2020
12
1
3
Hey guys,

I have currently a problem. My Internet in my Home-Office is slow. Here I get max 100 MBit down and 10MBit up. But most of my work, I have in the Datacenter on a TrueNAS ZFS. I tried to solve the problem with a Home-Server. To have some kind of files at home (for example static data like Intros, Footage) and to sync the working Data with a sync service.

To Sync files I tried Syncthing, it didn't work for me very well. Then I tried Resilio. It worked with normal files very well. With large files, it had some problems, but the most issues I got with a large number of small files. Like npm / node Packages. Resilio often deleted some of the files right under my butt.

Another problem was Unraid for me, with the one "cache" drive / nvme. As long as I have the Data on one fast NVME drive, when the drive dies, the data is goneski. When I run the mover hourly, and write large files to the Fileserver, the written files are moved away, as well as the directory. All of this is Crap.

What I'm looking is some kind of cache for my fileserver. Something like the ZIL / SLOG layer for my Fileserver. I want to see everything, that is on my Server in the Datacenter. But if I want to get a file from there, if it is on my cache Server, the Data will be transferred from there. If not, then from the Server in the Datacenter. Something like IPFS is doing already.

Does anyone has a solution?
 

Sean Ho

seanho.com
Nov 19, 2019
768
352
63
Vancouver, BC
seanho.com
I empathise with having constrained home internet; I lived with 30/6 DOCSIS for many years.

What are your primary use cases? You mention intros and footage, but also npm. Video editing and node dev have very different storage needs. I would do video editing completely at home, active projects on local NVMe and archived projects on spinners (a la Unraid), and just push the final renders to remote servers for publication. Proxy media generated by your video editing software are also handy (complete catalog of clips on NAS; proxy media on NVMe local to the editing workstation).
 

LL0rd

New Member
Feb 25, 2020
12
1
3
What are your primary use cases?
Well, on my storage I do two things:

1) I have my YouTube channel, where I publish once a week some videos. Most of the videos I shoot at home. So the primary source of the large files is at home. The editing is mostly done in the office. So the files has to go in some way to the office. But... it can happen, that there is a problem with a video. In this case I need the project file for the Video at home. So I can change something, render the video out and upload it.

2) Similar thing is with coding. But here I have my main Desktop in my office, my Desktop at home and a Notebook. My goal is to work seamlessly. The file, that I saved right now has to be few seconds (ok, minutes is also acceptable) later in the project directory on the other device. Partial synchronization (to exclude the node_modules directory) is not OK for me.
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
The classical solution for this issue is VDI, this way, you're not sending back and forth large files over chatty SMB protocol but processing all of it inside the data center, only sending video and control feeds. If your connection is, as you said, 100/10 Mbps, that should be plenty to use for either remote desktop protocol (HDX, Blast, PCoIP, etc...). Windows native RDP isn't a great fit for this.
 

Sean Ho

seanho.com
Nov 19, 2019
768
352
63
Vancouver, BC
seanho.com
For video editing, you might consider getting sufficient hardware so you can edit primarily at home, where the raw clips are stored.

For coding, this is a common use case, and you have many options depending on your workflow. VDI / remote desktop is one solution used by many. VSCode remote dev container is another way; editing stays local and low-latency, but the code runtime is all remote.

Another approach is to adjust your workflow around topic branches on a pull request associated with one or more issues/tickets. You can commit (and test) frequently to the work-in-progress topic branch and push/pull to various workstations as needed, then when it's ready for testing/staging releases, tidy up the changes and squash-commit, so your main trunk has just a single, clean commit.