scp -r --- Is my old habit terrible?

[Jeggs101]

I have a Linux server with the sole purpose of being a file server. It runs CentOS. It doesn't have any file services. Instead, I just

scp -r username@hostname:/path/to/dir .

then one of about 20 directories for the task I am doing.

I've been doing this for 7+ years so I'm accustomed to it. The only thing I don't really like is that it's on a separate server (not a NAS) and it's easy. The only part I'm not loving is that the hostname is long and there's minimal chance to shorten.

Does anyone else do this? It's gotta be world's worst admin habit.

 

[whitey]

scp is tried and true
rsync -avz (rsync over ssh as well if you set it up right)
netcat send receive switches (tcp/ip swiss army knife)
nfs/smb/iscsi (std network protocols/services)

Pick your poison, for small directories and daily tasks I use scp -r or sftp client (openssh across *nix systems) all the time. Nothing wrong w/ it, right tool for the right job I guess it boils down to.

 

[Blinky 42]

Nothing wrong with it by itself - I must type "rsync -avrmHz --progress --partial blah@foo:/..." 30 times on a slow day. I usually stick to rsync because it doesn't waste time copying what I have already and I can restart it if needed with those options.

Now if you are always pulling the same set of files form one server for some reason and it is something that should be in some sort source control then possibly tweak for that but otherwise don't fix what isn't broken. Just alias it to something with your shell or wrap it in a shell script if the server name is crazy (or CNAME it in DNS, add it to hosts).

 

[TuxDude]

Unless I'm only copying a single file, I pretty much universally use rsync instead of scp (or cp for that matter too, and also sometimes rsync/rm instead of mv). Takes a little bit of getting used to rsync's syntax, pay careful attention to whether or not you use a trailing / on directories (changes the meaning of whether you are copying the directory, or everything in the directory) - check the man page for examples and such. My typical command looks something like 'rsync -avhP --stats user@host:/..../ ..../', occasionally with a '-n' option added to run it in pretend mode and just see what it would do / how much data is going to be transferred.

@Blinky 42 - you can save a bit of typing, replacing '--progress --partial' with just a 'P' added to your first list of options, also '-r' is implied by '-a'. So you should get identical results with just 'rsync -avmHPz ...'

 

[chilipepperz]

I don't think scp is that bad of a habit.

 

[Blinky 42]

Yes @TuxDude is totally correct on optimal minimal typing.. the redundant long-form just evolved over the years as I mentally added more options on when typing it out by habit. Adding --partial is super handy transferring multi GB files around. Realizing you have to wait *again* for 20G of download on a slow network when it craps out at 99% done and you have to start from zero adjusts your default invocation after only a few times being burned.

I would suggest all *nix admins have a good working knowledge of wget, curl, rsync and scp to cover your file transfer needs. ncftp is handy as well but dropped out of common install at some point in the past even though it is a great as an interactive or scriptable client.
Doing tar or dd over ssh also is a useful trick that doesn't come up often but is damn handy when needed (live cd's and other odd rescue / CYA situations).

 

[TuxDude]

My most unusual file transfer requirement was pulling some VM images off a co-located all-in-one ESXi box that had only minimal remote access setup into the pfSense gateway running on it, and where said gateway didn't have nearly enough disk space to hold a temporary copy of the images. Thats when you come up with interesting solutions like:

ssh root@remote.gateway "wget --no-check-certificate 'https://user:password@esxi-internal-ip/folder/VM-Name/VM-Name.vmdk?dcPath=ha-datacenter&dsName=LOCAL' -q0 - | gzip -c" > VM-Name.vmdk.gz

 

[whitey]

That do a ssh to pfSense GW, run wget from there to ESXi hypervisor and redirect output to local machine that DOES have enough space to store image over network conn/tunnel?

LAN or WAN friendly (WAN taking longer of course unless you have a FAT circuit)? Assuming VM has to be shut down right for that to 'play nice'?

 

[TuxDude]

That do a ssh to pfSense GW, run wget from there to ESXi hypervisor and redirect output to local machine that DOES have enough space to store image over network conn/tunnel?

LAN or WAN friendly (WAN taking longer of course unless you have a FAT circuit)? Assuming VM has to be shut down right for that to 'play nice'?

Close. It does an SSH to the gateway, where it runs wget against the ESXi hypervisor. But the wget output is sent to stdout instead of a file (think output to your screen if you were executing it interactively), which is piped through gzip. So the "output" of the ssh command is the gzip'd data that was wget'ed, which at the local machine with lots of disk space is finally piped into a file. So the only part of your explanation that was off was that the pfSense gateway compresses the image before sending it over the WAN - its about as WAN friendly as possible, not that moving VM images over a WAN is ever really "friendly".

And no - the VM does not need to be shut down, which is the other WAN-friendly part of that method, though the VMDK files do need to be read-only when it is run. When I was doing that, my method was to take a VMware snapshot of the VM, forcing the main VMDK read-only and redirecting writes into a -00001.vmdk delta file. I could then download the base disk while leaving the VM running. You have two options after that point - 1. delete the snapshot, the VM continues to run and you've gotten yourself a "hot-clone" or "crash-consistent backup" on your local machine. or 2. In the future shut down the VM, and pull down the '-00001.vmdk' delta file using the same method, and a new/fresh copy of the .vmx file that references the correct vmdks - that allows you to pre-download the bulk of the VM in advance, and only require the VM be powered off for a short time later while you transfer the delta-file. You can then import the VM into a new ESXi and start it back up, and then consolidate the snapshot back into a single disk - makes for much faster offline VM migrations between hosts, needing to only transfer a few hundred MB's of delta instead of multi-GB images.

 

[fractal]

I have a Linux server with the sole purpose of being a file server. It runs CentOS. It doesn't have any file services. Instead, I just

scp -r username@hostname:/path/to/dir .

then one of about 20 directories for the task I am doing.

I've been doing this for 7+ years so I'm accustomed to it. The only thing I don't really like is that it's on a separate server (not a NAS) and it's easy. The only part I'm not loving is that the hostname is long and there's minimal chance to shorten.

Does anyone else do this? It's gotta be world's worst admin habit.

There is nothing wrong with scp. I often use scp for small jobs. I usually use rsync for large jobs.

I recently fell in love with ~/.ssh/config

I create an entry in it like:

Host file-host
  Hostname this_is_the_really_long_host_name._this_is_the_domain
  User this_is_my_user

which lets me shorten scp -r this_is_my_user&this_is_the_really_long_host_name._this_is_the_domain:/path/to/dir .

to

scp -r file-host:/path/to/dir .

One extra free benefit is I can hide jump hosts in the config file and not have to worry about how many intermediaries need to forward the packets.