Recursive download with pysftp

Question

I'm trying to fetch from SFTP with the following structure:

main_dir/
 dir1/
  file1
 dir2/
  file2

I tried to achieve this with commands below:

sftp.get_r(main_path + dirpath, local_path)

or

sftp.get_d(main_path + dirpath, local_path)

The local path is like d:/grabbed_files/target_dir , and the remote is like /data/some_dir/target_dir .

With get_r I am getting FileNotFound exception. With get_d I am getting empty dir (when target dir have files not dirs, it works fine).

I'm totally sure that directory exists at this path. What am I doing wrong?

Answer 1

This one works for me, but when you download directory it create full path locally.

pysftp.Connection.get_r()

I also created simple download and upload methods:

def download_r(sftp, outbox):
    tmp_dir = helpers.create_tmpdir()
    assert sftp.isdir(str(outbox))
    assert pathlib.Path(tmp_dir).is_dir()
    sftp.get_r(str(outbox), str(tmp_dir))
    tmp_dir = tmp_dir / outbox
    return tmp_dir


def upload_r(sftp, inbox, files):
    assert sftp.isdir(str(inbox))
    if pathlib.Path(files).is_dir():
        logger.debug(list(files.iterdir()))
        sftp.put_r(str(files), str(inbox))
    else:
        logger.debug('No files here.')

Answer 2

I didn't understand why it doesn't work so I ended with my own recursive solution:

def grab_dir_rec(sftp, dirpath):
    local_path = target_path + dirpath
    full_path = main_path + dirpath
    if not sftp.exists(full_path):
        return
    if not os.path.exists(local_path):
        os.makedirs(local_path)

    dirlist = sftp.listdir(remotepath=full_path)
    for i in dirlist:
        if sftp.isdir(full_path + '/' + i):
            grab_dir_rec(sftp, dirpath + '/' + i)
        else:
            grab_file(sftp, dirpath + '/' + i)

Answer 3

In the event that you want a context manager wrapper around pysftp that does this for you, here is a solution that is even less code (after you copy/paste the github gist) that ends up looking like the following when used

path = "sftp://user:password@test.com/path/to/file.txt"

# Read a file
with open_sftp(path) as f:
    s = f.read() 
print s

# Write to a file
with open_sftp(path, mode='w') as f:
    f.write("Some content.") 

The (fuller) example: http://www.prschmid.com/2016/09/simple-opensftp-context-manager-for.html

This context manager happens to have auto-retry logic baked in in the event you can't connect the first time around (which surprisingly happens more often than you'd expect in a production environment...).

Oh, and yes, this assumes you are only getting one file per connection as it will auto-close the ftp connection.

The context manager gist for open_sftp: https://gist.github.com/prschmid/80a19c22012e42d4d6e791c1e4eb8515