Cloud to Cloud file transfer Open source solution or private install needed

[DataBill]

I am looking for a solution similar to what Manage, Move, Copy, and Migrate Files Between Cloud Storage Services with MultCloud offers.

I never provide 3rd parties with access to my passwords or my data. As such I am trying to find an open source solution, or even a reasonably priced software package that would allow me to transfer files from one cloud service to another. I also need to be able to use a system that can monitor ftp servers and when the the content changes or is updated on these ftp servers it would automatically upload the changes to one of my cloud hosting services.

The system can run on any version of linux or windows as my servers operate under Xenserver.

 

[OBasel]

Sorry but what's the use case here? Are you doing server backups, PC backups?

 

[DataBill]

The use case is that I want to copy files from one could service to another. Or from hundreds of subscription FTP servers to cloud storage.

There are many services out there that can be used. The only trouble is one has to supply the account login details for everything they want to copy/move etc to another service. This means having to trust 100% these third parties with your highly sensitive login details not too mention the actual data itself.

I just came across Allway Sync (Allway Sync: Free File Synchronization, Backup, Data Replication, PC Sync Software, Freeware, File Sync, Data Synchronization Software which I am testing now. The program allows me to perform a sync in any manner. The advantage to this system is that all user credentials are stored locally. The only limitation I see so far will be based upon my own bandwidth speeds and bandwidth limits.

Basically in a nutshell there are three options as I see it to sync external data sources.

1. Download everything to your pc/server, then upload to the location you want it. The disadvantage to this process is that you are not only using double bandwidth but also using disk space on your transfer station when you do not need the information stored on it.

2. Use an online service which requires you to have 100% trust and faith in their security. This type of service can actually violate the TOS/Security of some subscriptions. Normally these types of services also charge by bandwidth used, which is far costlier than using a dedicated servers own bandwidth.

3. Self hosted script/software package. Advantages are that you do not need to trust anyone with your login credentials. The system does not actually store the data on your server either, it is merely a transfer point. The Cons:Uses double bandwidth for each file since you are downloading then uploading the same file.

All 3 options use double bandwidth, but option 3 is the most secure since it is totally controlled by the users system.

I am interested in knowing about all options and alternatives that use Option 3.

 

[MiniKnight]

We did something like this recently. The other consideration we had was that you never know what these service providers are doing. It'd be easy for them to filter and store any data they are transferring.

I'd do #3 but get 1-2 VPS's or dedicated servers with enough bandwidth. Many of the google, MSFT, S3 are all slower than your server anyway.