How to read contents and attributes of files in remote SFTP server using pysftp/Paramiko

Question

I am using below code to get list of file objects in list from two different nodes. My purpose is to get the file metadata and compare the files later. But using filelist = mysftp.listdir() , I am only getting the filenames and unable to fetch something like file.content or file.size .

For testing purpose I am using only one EC2 node and calling it parallel.

import pysftp
import itertools 
from multiprocessing import Process, Pool

# Connect/ssh to an instance

file_list = []

def readFileNode(host):
    try:
        cnopts = pysftp.CnOpts()
        cnopts.hostkeys = None

        mysftp = pysftp.Connection(
            host="3.93.XX.XX",
            username="ubuntu",
            private_key="/Users/test.pem",
            cnopts=cnopts,
        )

        mysftp.cwd("/home/ubuntu/demo")
        filelist = mysftp.listdir()

    except Exception:
        print(e)

    return filelist

if __name__ == '__main__':
    p = Pool(2)
    result = p.map(readFileNode, ['3.93.XX.XX', '3.93.XX.XX'])
    print(result)  

    print(result[0])
    print(result[1])
    fileList1=[]
    fileList2=[]
    fileList1=list[0]
    fileList2=list[1]
    
    for (a, b) in zip(fileList1,fileList2):
       print (a, b)
       #compare two files and also get the file metadata

I have files from both the nodes in result[0] and result[1] . But, I want to get the file objects so that I can compare them and list down the attributes. ie something like file.size , compare file1.content with file2.content . Where file1 is from node1 and file2 is from node2.

Answer 1

To retrieve file attributes, use Connection.listdir_attr instead of Connection.listdir .

To read the contents, use Connection.open .

for i in mysftp.listdir_attr(path):
    print i.filename + " " + i.st_mtime + " " + i.st_size
    with mysftp.open(path, 'r', 32768) as f:
        # Here you can use `f` as if it were a local file opened with `os.open`

---

Additionally, do not set the cnopts.hostkeys = None , unless you do not care about security.
For the correct solution, see Verify host key with pysftp .

---

Though your better use Paramiko instead of pysftp .

See How to fetch sizes of all SFTP files in a directory through Paramiko .