如何尽快复制文件？

Question

I am running my shell script on machineA which copies the files from machineB and machineC to machineA .

If the file is not there in machineB , then it should be there in machineC for sure. So I will try to copy from machineB first, if it is not there in machineB then I will go to machineC to copy the same files.

In machineB and machineC there will be a folder like this YYYYMMDD inside this folder -

/data/pe_t1_snapshot

So whatever date is the latest date in this format YYYYMMDD inside the above folder - I will pick that folder as the full path from where I need to start copying the files -

so suppose if this is the latest date folder 20140317 inside /data/pe_t1_snapshot then this will be the full path for me -

/data/pe_t1_snapshot/20140317

from where I need to start copying the files in machineB and machineC . I need to copy around 400 files in machineA from machineB and machineC and each file size is 1.5 GB .

Currently I have my below shell script which works fine as I am using scp but somehow it takes ~ 2 hours to copy the 400 files in machineA which is too long for me I guess. :(

Below is my shell script -

#!/bin/bash

readonly PRIMARY=/export/home/david/dist/primary
readonly SECONDARY=/export/home/david/dist/secondary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot
PRIMARY_PARTITION=(0 3 5 7 9) # this will have more file numbers around 200
SECONDARY_PARTITION=(1 2 4 6 8) # this will have more file numbers around 200

dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    # delete all the files first
    find "$PRIMARY" -mindepth 1 -delete
    for el in "${PRIMARY_PARTITION[@]}"
    do
        scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
    done

    # delete all the files first
    find "$SECONDARY" -mindepth 1 -delete
    for sl in "${SECONDARY_PARTITION[@]}"
    do
        scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.
    done
fi

I am copying PRIMARY_PARTITION files in PRIMARY folder and SECONDARY_PARTITION files in SECONDARY folder in machineA .

Is there any way to move the files faster in machineA . Can I copy 10 files at a time or 5 files at a time in parallel to speed up this process or any other approach?

NOTE: machineA is running on SSD

UPDATE:-

Parallel Shell Script which I tried, top portion of shell script is same as shown above.

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    find "$PRIMARY" -mindepth 1 -delete
    for el in "${PRIMARY_PARTITION[@]}"
    do
        (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.) &
          WAITPID="$WAITPID $!"        
    done

    find "$SECONDARY" -mindepth 1 -delete
    for sl in "${SECONDARY_PARTITION[@]}"
    do
        (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.) &
          WAITPID="$WAITPID $!"        
    done
     wait $WAITPID
     echo "All files done copying."
fi

Errors I got with parallel shell script-

channel 24: open failed: administratively prohibited: open failed
channel 25: open failed: administratively prohibited: open failed
channel 26: open failed: administratively prohibited: open failed
channel 28: open failed: administratively prohibited: open failed
channel 30: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
channel 32: open failed: administratively prohibited: open failed
channel 36: open failed: administratively prohibited: open failed
channel 37: open failed: administratively prohibited: open failed
channel 38: open failed: administratively prohibited: open failed
channel 40: open failed: administratively prohibited: open failed
channel 46: open failed: administratively prohibited: open failed
channel 47: open failed: administratively prohibited: open failed
channel 49: open failed: administratively prohibited: open failed
channel 52: open failed: administratively prohibited: open failed
channel 54: open failed: administratively prohibited: open failed
channel 55: open failed: administratively prohibited: open failed
channel 56: open failed: administratively prohibited: open failed
channel 57: open failed: administratively prohibited: open failed
channel 59: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
channel 61: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
channel 64: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
channel 68: open failed: administratively prohibited: open failed
channel 72: open failed: administratively prohibited: open failed
channel 74: open failed: administratively prohibited: open failed
channel 76: open failed: administratively prohibited: open failed
channel 78: open failed: administratively prohibited: open failed

Answer 1

you can try this command

rsync

from the

man rsync

you will see that: The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.

Answer 2

You may try the HPN-SSH (High Performance SSH/SCP) - http://www.psc.edu/index.php/hpn-ssh or http://hpnssh.sourceforge.net/

The HPN-SSH project is the set of patches for OpenSSH (scp is part of it), to better tune various tcp and internal buffers. There is also "none" cipher ("None Cipher Switching") which disables encryption, and this may help you too (if you don't use public networks to send the data).

Both compression and encryption consumes CPU time; and 10 Gbit Ethernet sometimes may be faster to transfer uncompressed file then waiting CPU to compress and encrypt it.

You may profile your setup:

• Measure the network bandwidth between machines using iperf or netperf . Compare with the actual network (network cards capabilities, switches). With good setup you should get more than 80-90 percents of declared speed.
• Calculate data volume and the time needed to transfer so much data with your network using speed from iperf or netperf . Compare with actual transfer time, is there huge difference?
  • If your CPU is fast, data is compressible and network is slow, compressing will help you.
• Take a look on top , vmstat , iostat .
  • Are there 100% loaded CPU cores (run top and press 1 to see cores)?
  • Are there too much interrupts ( in ) in vmstat 1 ? What about context switches ( cs )?
  • What is file reading speed in iostat 1 ? Are your HDDs are fast enough to read data; to write data on receiver?
• You can try to do full-system profiling using perf top or perf record -a . Is there lot of computing by scp, or network stack in Linux? If you can install dtrace or ktap , try to make also off-cpu profiling

Answer 3

You have 1.5 GB * 400 = 600 GB of data. Unrelated to the answer I suggest that the machine set up looks incorrect if you need to transfer this amount of data. You probably needed to generate this data at machine A in the first place.

There are 600 GB of data being transferred in 2 hours, that is ~ 85 MB/s transfer rate, which means you probably reached the transfer limits of either your disk drives or (almost) the network. I believe you won't be able to transfer faster with any other command.

If the machines are close to each other, the method of copying that I believe is the fastest is to physically remove the storage from machines B and C, put them in machine A and then locally copy them without transferring via the network. The time for this is the time to move around the storage, plus disk transfer times. I'm afraid, however, the copy won't be much faster than 85 MB/s.

The network transfer command that I believe would be the fastest one is netcat, because it has no overhead related to encryption. Additionally, if the files are not media files, you have to compress them using a compressor that compresses faster than 85 MB/s. I know of lzop and lz4 that are granted to be faster than this rate. So my command line for transfering a single directory would be (BSD netcat syntax):

machine A:

$ nc -l 2000 | lzop -d | tar x

machine B or C (can be executed from machine A with the help of ssh):

$ tar c directory | lzop | nc machineA 2000

Remove the compressor if transfering media files, which are already compressed.

The commands to organize your directory structure are irrelevant in terms of speed, so I didn't bother to write them here, but you can reuse your own code.

This is the fastest method I can think of, but, again, I don't believe this command will be much faster that what you already have.

Answer 4

You definitely want to give rclone a try. This thing is crazy fast :

sudo rclone sync /usr /home/fred/temp -P -L --transfers 64

Transferred: 17.929G / 17.929 GBytes, 100%, 165.692 MBytes/s, ETA 0s Errors: 75 (retrying may help) Checks: 691078 / 691078, 100% Transferred: 345539 / 345539, 100% Elapsed time: 1m50.8s

This is a local copy from and to a LITEONIT LCS-256 (256GB) SSD.

Answer 5

rsync optionally compresses its data. That typically makes the transfer go much faster.

You didn't mention SCP, but SCP -C also compresses.

Do note that compression might make the transfer go faster or slower, depending upon the speed of your CPU and of your network link.

Slower links and faster CPU make compression a good idea; faster links and slower CPU make compression a bad idea.

As with any optimization, measure the results in your own environment.

Also I think ftp is another option for you, as my transfer speed test for large files (>10M) FTP work faster then SCP and even rsync (It's depended on file format and compression rate).

Answer 6

rsync is a good answer, but if you care about security then you should consider using:

rdist

Some details on the differences between rsync and rdist can be found here: rdist vs rsync and a blog about how to set it up using ssh can be found here: non root remote updating

Finally you could use the infamous tar pipe tar pattern, with a sprinkle of ssh.

tar zcvf - /wwwdata | ssh root@dumpserver.nixcraft.in "cat > /backup/wwwdata.tar.gz"

This example is talked about here: tar copy over secure network

Answer 7

The remote doesn't support ssh multiplexing.

To silence the message:

mux_client_request_session: session request failed: Session open refused by peer

Change your ~/.ssh/config file:

Host destination.hostname.com
  ControlMaster no

Host *
  ControlMaster auto
  ControlPersist yes
  ControlPath ~/.ssh/socket-%r@%h:%p

More details and notes can be found here .