1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

scp -r --- Is my old habit terrible?

Discussion in 'Linux Admins, Storage and Virtualization' started by Jeggs101, Jan 9, 2017.

  1. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,073
    Likes Received:
    122
    I have a Linux server with the sole purpose of being a file server. It runs CentOS. It doesn't have any file services. Instead, I just
    Code:
    scp -r username@hostname:/path/to/dir .
    then one of about 20 directories for the task I am doing.

    I've been doing this for 7+ years so I'm accustomed to it. The only thing I don't really like is that it's on a separate server (not a NAS) and it's easy. The only part I'm not loving is that the hostname is long and there's minimal chance to shorten.

    Does anyone else do this? It's gotta be world's worst admin habit.
     
    #1
  2. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    scp is tried and true
    rsync -avz (rsync over ssh as well if you set it up right)
    netcat send receive switches (tcp/ip swiss army knife)
    nfs/smb/iscsi (std network protocols/services)

    Pick your poison, for small directories and daily tasks I use scp -r or sftp client (openssh across *nix systems) all the time. Nothing wrong w/ it, right tool for the right job I guess it boils down to.
     
    #2
  3. Blinky 42

    Blinky 42 Active Member

    Joined:
    Aug 6, 2015
    Messages:
    229
    Likes Received:
    72
    Nothing wrong with it by itself - I must type "rsync -avrmHz --progress --partial blah@foo:/..." 30 times on a slow day. I usually stick to rsync because it doesn't waste time copying what I have already and I can restart it if needed with those options.

    Now if you are always pulling the same set of files form one server for some reason and it is something that should be in some sort source control then possibly tweak for that but otherwise don't fix what isn't broken. Just alias it to something with your shell or wrap it in a shell script if the server name is crazy (or CNAME it in DNS, add it to hosts).
     
    #3
  4. TuxDude

    TuxDude Well-Known Member

    Joined:
    Sep 17, 2011
    Messages:
    609
    Likes Received:
    330
    Unless I'm only copying a single file, I pretty much universally use rsync instead of scp (or cp for that matter too, and also sometimes rsync/rm instead of mv). Takes a little bit of getting used to rsync's syntax, pay careful attention to whether or not you use a trailing / on directories (changes the meaning of whether you are copying the directory, or everything in the directory) - check the man page for examples and such. My typical command looks something like 'rsync -avhP --stats user@host:/..../ ..../', occasionally with a '-n' option added to run it in pretend mode and just see what it would do / how much data is going to be transferred.

    @Blinky 42 - you can save a bit of typing, replacing '--progress --partial' with just a 'P' added to your first list of options, also '-r' is implied by '-a'. So you should get identical results with just 'rsync -avmHPz ...'
     
    #4
    dawsonkm, Patriot and chilipepperz like this.
  5. chilipepperz

    chilipepperz Member

    Joined:
    Mar 17, 2016
    Messages:
    63
    Likes Received:
    17
    I don't think scp is that bad of a habit.
     
    #5
  6. Blinky 42

    Blinky 42 Active Member

    Joined:
    Aug 6, 2015
    Messages:
    229
    Likes Received:
    72
    Yes @TuxDude is totally correct on optimal minimal typing.. the redundant long-form just evolved over the years as I mentally added more options on when typing it out by habit. Adding --partial is super handy transferring multi GB files around. Realizing you have to wait *again* for 20G of download on a slow network when it craps out at 99% done and you have to start from zero adjusts your default invocation after only a few times being burned.

    I would suggest all *nix admins have a good working knowledge of wget, curl, rsync and scp to cover your file transfer needs. ncftp is handy as well but dropped out of common install at some point in the past even though it is a great as an interactive or scriptable client.
    Doing tar or dd over ssh also is a useful trick that doesn't come up often but is damn handy when needed (live cd's and other odd rescue / CYA situations).
     
    #6
  7. TuxDude

    TuxDude Well-Known Member

    Joined:
    Sep 17, 2011
    Messages:
    609
    Likes Received:
    330
    My most unusual file transfer requirement was pulling some VM images off a co-located all-in-one ESXi box that had only minimal remote access setup into the pfSense gateway running on it, and where said gateway didn't have nearly enough disk space to hold a temporary copy of the images. Thats when you come up with interesting solutions like:

    Code:
    ssh root@remote.gateway "wget --no-check-certificate 'https://user:password@esxi-internal-ip/folder/VM-Name/VM-Name.vmdk?dcPath=ha-datacenter&dsName=LOCAL' -q0 - | gzip -c" > VM-Name.vmdk.gz
     
    #7
    whitey, Blinky 42 and cperalt1 like this.
  8. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    That do a ssh to pfSense GW, run wget from there to ESXi hypervisor and redirect output to local machine that DOES have enough space to store image over network conn/tunnel?

    LAN or WAN friendly (WAN taking longer of course unless you have a FAT circuit)? Assuming VM has to be shut down right for that to 'play nice'?
     
    #8
  9. TuxDude

    TuxDude Well-Known Member

    Joined:
    Sep 17, 2011
    Messages:
    609
    Likes Received:
    330

    Close. It does an SSH to the gateway, where it runs wget against the ESXi hypervisor. But the wget output is sent to stdout instead of a file (think output to your screen if you were executing it interactively), which is piped through gzip. So the "output" of the ssh command is the gzip'd data that was wget'ed, which at the local machine with lots of disk space is finally piped into a file. So the only part of your explanation that was off was that the pfSense gateway compresses the image before sending it over the WAN - its about as WAN friendly as possible, not that moving VM images over a WAN is ever really "friendly".

    And no - the VM does not need to be shut down, which is the other WAN-friendly part of that method, though the VMDK files do need to be read-only when it is run. When I was doing that, my method was to take a VMware snapshot of the VM, forcing the main VMDK read-only and redirecting writes into a -00001.vmdk delta file. I could then download the base disk while leaving the VM running. You have two options after that point - 1. delete the snapshot, the VM continues to run and you've gotten yourself a "hot-clone" or "crash-consistent backup" on your local machine. or 2. In the future shut down the VM, and pull down the '-00001.vmdk' delta file using the same method, and a new/fresh copy of the .vmx file that references the correct vmdks - that allows you to pre-download the bulk of the VM in advance, and only require the VM be powered off for a short time later while you transfer the delta-file. You can then import the VM into a new ESXi and start it back up, and then consolidate the snapshot back into a single disk - makes for much faster offline VM migrations between hosts, needing to only transfer a few hundred MB's of delta instead of multi-GB images.
     
    #9
    whitey, Patrick and _alex like this.
  10. fractal

    fractal Active Member

    Joined:
    Jun 7, 2016
    Messages:
    215
    Likes Received:
    43
    There is nothing wrong with scp. I often use scp for small jobs. I usually use rsync for large jobs.

    I recently fell in love with ~/.ssh/config

    I create an entry in it like:
    Code:
    Host file-host
      Hostname this_is_the_really_long_host_name._this_is_the_domain
      User this_is_my_user
    
    which lets me shorten scp -r this_is_my_user&this_is_the_really_long_host_name._this_is_the_domain:/path/to/dir .

    to

    scp -r file-host:/path/to/dir .

    One extra free benefit is I can hide jump hosts in the config file and not have to worry about how many intermediaries need to forward the packets.
     
    #10

Share This Page