rsync only files younger than xy days

Question

I am using rsync to copy photos form our satellite servers into main server. So the script doing it is basically connecting from PC to PC and executing rsync .

I have been trying to use find to determine files younger than xy days (it will be days, but number can vary). Specifing the files with --files-from=<() BUT the command find /var/dav/davserver/ -mtime -3 -type f -exec basename {} \\; is on some machines very very slow, and even makes rsync to timeout. Also they are servers, so running this command every few minutes would cost too much processor power that I don't want to take away.

The second aproach was to take advantage of the way we are storing those files, under /var/dav/davserver/year/month/day/ directory. However as I started to work on it, I have realized that I need to write quite a some code do take care of end of months and years, even more that number of days is not fixed (it can be more than 31 days, thus this scrip could need to run through several months).

So I was wondering if there is not some easier way how to achieve this without killing source PCs processor or write a whole new library to take care of all month/year ends?

---

EDIT:

I have prepared script that generates paths to files for me. What I did, is that I left handling end of months/year for date ..

#!/bin/bash

DATE_now=`date +"%Y-%m-%d"`
DATE_end=`date -d "-$1 days" +"%Y-%m-%d"`

echo "Date now: $DATE_now | Date end: $DATE_end"

start_d=`date +%s`
end_d=`date -d "-$1 days" +%s`

synced_day=$DATE_now
synced_day_s=$start_d
daycount=1

echo "" > /tmp/$2_paths

while [ $synced_day_s -ge $end_d ]; do
    DAY=$(date -d "$synced_day" '+%d')
    MONTH=$(date -d "$synced_day" '+%m')
    YEAR=$(date -d "$synced_day" '+%Y')

    SYNC_DIR="/var/dav/davserver/$YEAR/$MONTH/$DAY/**"
    echo "Adding day ($synced_day) directory: \"$SYNC_DIR\" to synced paths | Day: $daycount"
    echo $SYNC_DIR >> /tmp/$2_paths

    synced_day=$(date -d "$synced_day -1 days" +"%Y-%m-%d")
    synced_day_s=$(date -d "$synced_day" +%s)
    daycount=$((daycount+1))
done

and counting down days using it, than just extract needed info. This script gives me a list of directories to rsync:

rrr@rRr-kali:~/bash_devel$ bash date_extract.sh 8 Z00163
Date now: 2017-06-29 | Date end: 2017-06-21
Adding day (2017-06-29) directory: "/var/dav/davserver/2017/06/29/**" to synced paths | Day: 1
Adding day (2017-06-28) directory: "/var/dav/davserver/2017/06/28/**" to synced paths | Day: 2
Adding day (2017-06-27) directory: "/var/dav/davserver/2017/06/27/**" to synced paths | Day: 3
Adding day (2017-06-26) directory: "/var/dav/davserver/2017/06/26/**" to synced paths | Day: 4
Adding day (2017-06-25) directory: "/var/dav/davserver/2017/06/25/**" to synced paths | Day: 5
Adding day (2017-06-24) directory: "/var/dav/davserver/2017/06/24/**" to synced paths | Day: 6
Adding day (2017-06-23) directory: "/var/dav/davserver/2017/06/23/**" to synced paths | Day: 7
Adding day (2017-06-22) directory: "/var/dav/davserver/2017/06/22/**" to synced paths | Day: 8
rrr@rRr-kali:~/bash_devel$ cat /tmp/Z00163_paths 

/var/dav/davserver/2017/06/29/**
/var/dav/davserver/2017/06/28/**
/var/dav/davserver/2017/06/27/**
/var/dav/davserver/2017/06/26/**
/var/dav/davserver/2017/06/25/**
/var/dav/davserver/2017/06/24/**
/var/dav/davserver/2017/06/23/**
/var/dav/davserver/2017/06/22/**
rrr@rRr-kali:~/bash_devel$

However, now is my problem to use this list, I have been trying to use many combination of --include and --exclude commands with both --include-files and --include-from BUT I am getting only 2 results: either everything is being rsynced, or nothing.

Answer 1

Since you already have files ordered by date (in directories), it's easy and efficient to just rsync those directories:

#!/bin/bash
maxage="45"  # in days, from today
for ((d=0; d<=maxage; d++)); do
    dir="/var/dav/davserver/$(date -d "-$d day" +"%Y/%m/%d")"
    rsync -avrz server:"$dir" localdir
done

We're using date to calculate today - x days and iterate over all days from 0 to your maxage .

Edit: use arithmetic for loop instead of iterating over GNU seq range.

Answer 2

So, I have solved it with combination of:

Script generating paths according to actual date. Details are presented in my edit of initial post. It simply uses date to go through previous days and manage month and year ends. And generate paths from those dates. However radomir's solution is simplier, so I will use it. (Its basically same as I did, just simplier way to write it down).

than I have used combination of --include-files=/tmp/files_list and -r aka --recursive argument to use this list of paths properly. (It was copying only empty directories without -r . And everything or nothing if I have used --include-from instead of --include-files )

Final rsync command is:

rsync --timeout=300 -Sazrv --force --delete --numeric-ids --files-from=/tmp/date_paths app_core@172.23.160.1:/var/dav/davserver/ /data/snapshots/

However this solution is not deleting old files on my side, despite --delete argument. Will probably need to make an extra script for it.