How to capture the output of sftp ls command from an SFTP server using expect?

Question

I've been struggling since the last three days to write a bash script which automates the downloading of files from an SFTP server. I've built the structure of the program, have tested it in snippets but this is what I'm stuck on.

I log into the SFTP server thus:

/usr/bin/expect <<EOD
spawn sftp $ftp_server
expect "password:"
send "$password\r"
expect "sftp>"
send "ls\r"
expect "sftp>\r"
send "exit\r"
EOD

I want to loop over the output of the ls command to decide which file to download. I tried redirecting the output to a text file and then picking up the file names from there, but it stores the "sftp>" prompts and other irrelevant information as well. How can I store the clean ls output of expect and loop over it?

Answer 1

Use -b switch to pass a script with the commands ( ls ), instead of feeding them in standard input.

This way the sftp will run in a batch mode without prompts.

Answer 2

There is example how to catch all filenames to list:

#!/bin/sh
# the next line restarts using expect \
    LC_TYPE=C exec expect -f "$0" -- "$@"

# do not show sftp output by default 
log_user 0

set ftp_server 127.0.0.1
set password pass
set sftp_prompt {sftp> }

spawn -noecho sftp $ftp_server

expect "password:"
send "$password\r"

expect $sftp_prompt

# 'ls -1' will show filenames line by line
send "ls -1\r"

# ignore echo of command from sftp
expect -re {ls -1\r?\n}

# init empty list for filename collecting
set files {}

expect -re {([^\r\n]+)\r?\n} {
    # catch each filename line by line and put it to list 'files'
    lappend files $expect_out(1,string)

    # run current 'expect' again to catch next filename
    exp_continue
} -ex $sftp_prompt {
    # catch sftp prompt to break
}

# loop over example :)
foreach f $files {
    puts $f
}