Upload a file-like object with Paramiko?

Question

I have a bunch of code that looks like this:

with tempfile.NamedTemporaryFile() as tmpfile:
    tmpfile.write(fileobj.read()) # fileobj is some file-like object
    tmpfile.flush()
    try:
        self.sftp.put(tmpfile.name, path)
    except IOError:
        # error handling removed for ease of reading
        pass

Is it possible to do an upload like this without having to write the file out somewhere?

Answer 1

Update As of Paramiko 1.10 , you can use putfo :

self.sftp.putfo(fileobj, path)

---

Instead of using paramiko.SFTPClient.put , you can use paramiko.SFTPClient.open , which opens a file -like object. You can write to that. Something like this:

f = self.sftp.open(path, 'wb')
f.write(fileobj.read())
f.close()

Note that it may be worthwhile to feed paramiko data in 32 KiB chunks, since that's the largest chunk underlying SSH protocol can handle without breaking it into multiple packets.

Answer 2

Is StringIO what you're looking for? ( doc page )

SFTPClient 's get() and put() functions take paths and not file-handles, which makes things a bit awkward.

You could write a wrapper for paramiko.SFTPClient to give it the functionality that you want.

Here's my best untested attempt:

from paramiko import SFTPClient

class SFTPClient2(SFTPClient):
    def put(self, local_file, remotepath, callback=None, confirm=True):
        fl = source_file
        file_size = os.fstat(fl.fileno()).st_size
        try:
            fr = self.file(remotepath, 'wb')
            fr.set_pipelined(True)
            size = 0
            try:
                while True:
                    data = fl.read(32768)
                    if len(data) == 0:
                        break
                    fr.write(data)
                    size += len(data)
                    if callback is not None:
                        callback(size, file_size)
            finally:
                fr.close()
        finally:
            fl.close()
        if confirm:
            s = self.stat(remotepath)
            if s.st_size != size:
                raise IOError('size mismatch in put!  %d != %d' % (s.st_size, size))
        else:
            s = SFTPAttributes()
        return s

    def get(self, remotepath, local_file, callback=None):
        fr = self.file(remotepath, 'rb')
        file_size = self.stat(remotepath).st_size
        fr.prefetch()
        try:
            fl = local_file
            try:
                size = 0
                while True:
                    data = fr.read(32768)
                    if len(data) == 0:
                        break
                    fl.write(data)
                    size += len(data)
                    if callback is not None:
                        callback(size, file_size)
            finally:
                fl.close()
        finally:
            fr.close()
        s = os.fstat(fl.fileno())
        if s.st_size != size:
            raise IOError('size mismatch in get!  %d != %d' % (s.st_size, size))

If it works, the get and put functions should now take local file-handles rather than paths.

All I had to do was get rid of the code that opens the file from the path, and change the code that gets the size of the file to use os.fstat instead of os.stat .