使用 pysftp 从 SFTP 读取 SHP 文件

Question

I am trying to use pysftp's getfo() to read a shapefile (without downloading it). However the output I get does not seem workable and I'm not sure if its possible do this with a shapefile.

Ideally I would like to read in the file and convert it to a Geopandas GeoDataFrame.

import pysftp
import io

with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword, cnopts=cnopts) as sftp:
    print("Connection established ... ")

    
   
    # check files in directory
    directory_structure = sftp.listdir('sites')
    

    # Print data
    for attr in directory_structure:
        print(attr)
    
    
    flo = io.BytesIO()
    sites = sftp.getfo('sites/Sites.shp', flo)
    value=flo.getvalue()
   

From here I can't decode the value and am unsure of how to proceed.

Answer 1

Something like this should do:

flo.seek(0)
df = geopandas.read_file(shp=flo)

---

Though using the Connection.getfo unnecesarily keeps whole raw file in memory. More efficient would be:

with sftp.open('sites/Sites.shp', bufsize=32768) as f:
    df = geopandas.read_file(f)

(for the purpose of bufsize=32768 , see Reading file opened with Python Paramiko SFTPClient.open method is slow )

---

Btw, note that the code downloads the file anyway. You cannot parse a remote file contents, without actually downloading that file contents. The code just avoids storing the downloaded file contents to a (temporary) local file.

Answer 2

Using Martin's answer, I was able to get to the outcome I needed. However it was required that the data was zipped in the SFTP, so it required a change to the datasource.

The output here is a dataframe but this is easily adapted to be a GeoDataFrame.

with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword, cnopts=cnopts) as sftp:
    print("Connection succesfully stablished ... ")  


    zipshape = zipfile.ZipFile(sftp.open('sites/sites_test.zip', bufsize=32768))
    r = shapefile.Reader(
            shp=zipshape.open('sites/Sites.shp'),
            shx=zipshape.open('sites/Sites.SHX'),
            dbf=zipshape.open('sites/Sites.DBF')
        )
    
    #check we have actually got the file
    print(r.bbox)
    print(r.numRecords)
    
    #get field names
    fields = [x[0] for x in r.fields][1:]
    records = r.records()
    
    #get coords and tidy up
    shps = [s.points for s in r.shapes()]
    
    #write the records into a dataframe 
    gdf = pd.DataFrame(columns=fields, data=records)

    #add the coordinate data to a column called "coords" 
    gdf = gdf.assign(coords=coords)