lftp delete multiples files with Bash

Question

I try to create a script who delete all the olds files except the three more recent files on my backup directory with lftp.

I have try to do this with ls -1tr who return all the files in ascending date order, and after I do a head -$NB_BACKUP_TO_RM ($NB_BACKUP_TO_RM is the numbers of files that I want to delete in my lists), this two commands return the correct files.

After this I want to remove all of them, so I do a xargs rm -- , but Bash returns that the files don't exist... I think this command is not running into the remote directory, but in the local directory, and I don't know what I can do for delete this files (of my return lists).

Here is the full code:

MAX_BACKUP=3
NB_BACKUP=$(lftp -e "ls -1tr $REMOTE_DIR/full_backup_ftp* | wc -l ; quit" -u $USER,$PASSWORD $HOST)

if (( $NB_BACKUP > $MAX_BACKUP ))
then
   NB_BACKUP_TO_RM=$(($NB_BACKUP-$MAX_BACKUP))
   REMOVE=$(lftp -e "ls -1tr $REMOTE_DIR/full_backup_ftp* | head -$NB_BACKUP_TO_RM | xargs rm -- ; quit" -u $USER,$PASSWORD $HOST)
   echo $REMOVE
fi

Have you an idea of the problem? How can I delete the files of my lists (after ls -1tr $REMOTE_DIR/full_backup_ftp* and head -$NB_BACKUP_TO_RM )

Thanks for your help

Answer 1

Starting SFTP connection can be time consuming. Slightly modified solution to avoid multiple lftp sessions below. It will perform much better the the alternative solution, especially if large number of files have to be purged.

Basically, leveraging lftp flexibility to mix lftp command with external commands. It creates a command file with a series of 'rm' (leveraging head,xargs, ...), and executing those commands INSIDE the same lftp session.

Also note that lftp 'ls' does not allow wildcard, use 'cls' instead

Make sure you test this carefully, because of potential removal of important files

lftp -e $USER,$PASSWORD $HOST <<__CMD__
   cls -1tr $REMOTE_DIR/full_backup_ftp* | head -$NB_BACKUP_TO_RM | xarg -I{} echo rm {} > rm_list.txt
   source rm_list.txt
__CMD__

Or with one liner, using 'lftp' ability to execute dynamically generated command (source -e). It eliminate the temporary file.

lftp -e $USER,$PASSWORD $HOST <<__CMD__
   source -e 'cls -1tr $REMOTE_DIR/full_backup_ftp* | head -$NB_BACKUP_TO_RM | xarg -I{} echo rm {}'
__CMD__

Answer 2

Looks xargs is unknown cmd for lftp after man lftp . And xargs rm is deleting local files not remote files.

so please use xargs as below, it works for me.

lftp -e "ls -1tr $REMOTE_DIR/full_backup_ftp*; quit" -u $USER,$PASSWORD $HOST | head -$NB_BACKUP_TO_RM | xargs -I {} lftp -e 'rm '{}'; quit' -u $USER,$PASSWORD $HOST