使用SCP大量传输文件

Question

I'm trying to transfer large quantities of data from one server to another. I've tried using

scp <file> <address@server:/path/to/scp/to

Which works, however it's quite slow for large files and requires that I enter a password for each scp.

I've also tried zipping and unzipping before and post transfer but it's even slower. Is there faster way?

Is this the best way to transfer large amounts of data?

Answer 1

rsync is a good option. It supports ssh as well as it support compression too. Check man rsync

Answer 2

If you want to compress and transfer all files in a directory you can use this:

tar -cf - <DIR>/* | sshpass -p '<password>' ssh <address@server>  'tar -xf - -C /path/to/unzip/to'

This will compress all files, transfer via ssh (auto-fill password) and unzip them in a specified path on your second server.

You could put this in a bash script as follows:

Show all dirs in cd and set as $mydir:

mydirs=`ls -p | grep "/"`

Execute script for each dir in cd:

for i in $mydirs; 
do 
    echo "Zipping directory ($i)...";
    tar -cf - <DIR>/* | sshpass -p '<password>' ssh <address@server>  'tar -xf - -C /path/to/unzip/to'
done

Answer 3

One common problem with SCP is that it doesn't pipeline requests, and so, it can be very slow when transferring lots of small files.

As @FlyingFrog has already sugested, that can be solved using tar .

From Perl you can do it easyly using Net::OpenSSH which can also handle the password authentication.

use Net::OpenSSH;
my $ssh = Net::OpenSSH->new($host, user => $user, password => $password);
$ssh->system({ stdin_file => ['-|', 'tar', 'czf', '-', @files] },
             "cd $dest && tar zzf - ");

Answer 4

I would recommend using http://www.ncftp.com/ . NcFTP server is what you need. It works fast and best when you have to transfer files between servers.