How do you guys move large amounts of data?

Fritz

Well-Known Member
Apr 6, 2015
2,151
508
113
66
I'm in the process of copying 11.4TB of data from one network location to another. It's all one operation but I was wondering if I'd gain any benefit from breaking it up into 2 or 3 batches. Thanks :)
 

Markus

Member
Oct 25, 2015
78
19
8
It depends (Protocol, Network...).
If you are using kind of bonded interfaces you can get more speed if you divide the transfer into multiple streams.
 

NetWise

Active Member
Jun 29, 2012
599
131
43
Edmonton, AB, Canada
For me, multiple robocopy streams, in multithreaded mode. Depends more on the number of files than size as to how I break it up.

Then just time. Lots and lots of time.


Sent from my iPhone using Tapatalk
 

Brian Puccio

Member
Jul 26, 2014
41
11
8
37
One bit at a time :)

Internally? rsync (and lately zfs send/receive) over directly connection 10gbe.

Externally? I'm limited by my cable connection for uploading to ACD but use rclone with encryption. I do push several 4x a day backup jobs from VPSes to a local fileserver (incremental, rolling) with attic.
 

Blinky 42

Active Member
Aug 6, 2015
568
203
43
44
PA, USA
Code:
rsync -avrmH --progress --partial
Is my go-to command for most situations. The advantage of stopping and restarting it as needed and having it print out where it is so you can wait 5 seconds to finish that big file before stopping it is worth it. Usually run in screen.
In situations where the source files only ever grow or static sized and there are not super security worries (like video recordings) I will set up a http server and mirror the content down using curl to avoid the overhead of encryption. Compression is worth it on the ssh connection if the data can be compressed and isn't already (VM images, oracle backups, etc).

When moving more than a few TB offsite, I'll frequently stick the data on portable USB drives and FedEx or DHL it. Ends up being cheaper and faster than the data overages at colos or tying up data pipes for days at a client's site + the drives end up being another backup if not reused. Also handy is --bwlimit to rsync if on a connection where I still want to use it for other things and just copy in the background.
 

cheezehead

Active Member
Sep 23, 2012
718
174
43
WI
Depends on the pipe and the source/target combination. If bandwidth is plenty then rsync/robocopy, if not then i'd ship a couple of drives and "seed" the data at the target then using rsync/robocopy to bring it current before cutting over.

I've also heard of some using BTSync 1.x, but have since moved on to Syncthing because of the BTSync 2.x paid model.
 

Jon Massey

Active Member
Nov 11, 2015
340
82
28
34
Sneakernet FTW. 4TB drive in a peli case, overnight courier is how I roll at work - http://i.imgur.com/LpLG9vz.jpg. One of our sites has awful bw so I get one of these every week. Other sites are better connected and I use RClone to B2.
 

capn_pineapple

Active Member
Aug 28, 2013
356
80
28
With the density and low physical size/weight of SSD's now, there's no reason that IPoAC isn't viable. Packet loss is still an issue but the bandwidth is sky high.
 
  • Like
Reactions: pgh5278

pgh5278

Active Member
Oct 25, 2012
474
127
43
Australia
good one..
Cisco even have documentation. Other documents seem to reveal it can be useful in war time, as difficult to hack. Note mentions max transmission distance is 1100 miles, bit further than the distance from Brisbane to Melbourne?, so we may not be able to test it. Do have a friend who has tried the system and can successfully ping from one station to one out of the city, but has not used for data transmission yet.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,287
439
83
With the density and low physical size/weight of SSD's now, there's no reason that IPoAC isn't viable. Packet loss is still an issue but the bandwidth is sky high.
Pie-in-the-sky thinking really, in the real world latency is a problem.
Code:
Script started on Sat Apr 28 11:24:09 2001
vegard@gyversalen:~$ /sbin/ifconfig tun0
tun0      Link encap:Point-to-Point Protocol  
          inet addr:10.0.3.2  P-t-P:10.0.3.1  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:150  Metric:1
          RX packets:1 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 
          RX bytes:88 (88.0 b)  TX bytes:168 (168.0 b)

vegard@gyversalen:~$ ping -i 900 10.0.3.1
PING 10.0.3.1 (10.0.3.1): 56 data bytes
64 bytes from 10.0.3.1: icmp_seq=0 ttl=255 time=6165731.1 ms
64 bytes from 10.0.3.1: icmp_seq=4 ttl=255 time=3211900.8 ms
64 bytes from 10.0.3.1: icmp_seq=2 ttl=255 time=5124922.8 ms
64 bytes from 10.0.3.1: icmp_seq=1 ttl=255 time=6388671.9 ms

--- 10.0.3.1 ping statistics ---
9 packets transmitted, 4 packets received, 55% packet loss
round-trip min/avg/max = 3211900.8/5222806.6/6388671.9 ms
vegard@gyversalen:~$ exit

Script done on Sat Apr 28 14:14:28 2001
 
  • Like
Reactions: capn_pineapple