Shell script to copy one file at a time in a cron job

Question

I have some csv files in location A like this

abc1.csv, abc2.csv, abc3.csv

I have a cron job which runs every 30 mins and in each execution I want to copy only 1 file(which shouldn't be repeated) and placed it in location B

I had though of 2 ways of doing this 1)I will pick the first file in the list of files and copy it to location B and will delete it once copied.Problem with this is I am not sure when the file will get copied completely and if i delete before its completed copied it can be an issue 2)I will have a temp folder.So i will copy the file from location A to location B and also keep it in temp location.In next iteration, when I pick the file from list of files I will compare its existence in the temp file location.If it exists I will move to next file.I think this will be more time consuming etc.

Please suggest if there is any other better way

Answer 1

You can ensure you move the already copied file with:

cp abc1.csv destination/ && mv abc1.csv.done

(here you can make your logic to find only *.csv files, and not take into account *.done files.. that have been already processed by your script... or use any suffix you want..

if the cp does not succeed, nothing after that will get executed, so the file will not be moved.

You can also replace mv with rm to delete it:

cp abc1.csv destination/ && rm -f abc1.csv

Further more, you can add to the above commands error messages in case you want to be informed if the cp failed:

cp abc1.csv destination/ && mv abc1.csv.done || echo "copy of file abc1.csv failed"

And get informed via CRON/email output

Answer 2

you can use this bash script for your use case:

source="/path/to/.csv/directory"
dest="/path/to/destination/directory"

cd $source

for file in *.csv
do
        if [ ! -f $dest/"$file" ]
        then
                cp -v $file $dest
                break
        fi
done

Answer 3

Finally I took some idea from both the opted solution.Here is the final script

source="/path/to/.csv/directory"
dest="/path/to/destination/directory"

cd $source

for file in *.csv
do
        if [ ! -f $dest/"$file" ]
        then
                cp -v $file $dest || echo "copy of file $file failed"
                rm -f $file
                break
        fi
done