Bash script: Looping through directories over SSH and downloading them to local machine

Question

I'm trying to write a bash script that:

• Loops through all the directories in a folder over an SSH server
• Downloads a file (titled say, "foo") inside each of these folders to a local machine.

At the moment, I have:

ssh username@server "for dir in ~/directoryname/*; (... something here!); done"

I don't think I can use scp while I'm accessing the SSH server, however. Is there a way I can loop through and download everything here?

Answer 1

running scp remotely would work only if the remote server has access to your own system. let's assume it doesn't.

you could do it in two steps:

ssh username@server "... some script that just echos the paths ..." > log
for line in $(<log); do scp username@server:$line ./dir/$line; done

or you could investigate rsync which is extremely powerful. it has --include and --exclude options which would allow you to do something like:

rsync -av username@server:~/somepath/ ./somepath/ [--exclude/--include flags]

Answer 2

Yeah, that script is an argument passed to ssh, and is run on the remote machine, which means you can't easily reference your own machine in order to tell it where to copy the files to.

You could try a bash script that runs locally to fetch the directory structure you're searching, and use scp to copy over any that you're interested in retrieving.

Or, depending on the sizes involved, scp over all of directoryname/ , and get what you want from there.