pandas：如何加快数据帧处理

Question

I am doing some distance processing on two dataframes (100k lines and 1M lines). My processing takes 20 days at the moment and I would like to see if I can improve my code to speed up the process. I used geopandas after a suggestion here, which considerably speeded up the sorting in my iteration, but i'm wondering if I could code it differently or some best practices.

Here is my code:

    dfb=pd.read_csv(building,sep='#')
    dfb=dfb.astype('string')
    dfo=pd.read_csv(occsol,sep='#')
    dfo=dfo.astype('string')
    dfb['geot']='non'
    gs = gpd.GeoSeries.from_wkt(dfo['geometry'], crs='EPSG:27572')
    gdfo=gpd.GeoDataFrame(dfo,geometry=gs)
    dfb['valeurseuil'] = 3 * ((dfb['surf'] / 3.141592653589793) ** (1 / 2))  # this is a treshold
    m = 0
    fin = len(dfb)
    for i in range(len(dfb)):  
        gdfo['dist']=gdfo['geometry'].distance(Point(dfb.iloc[i]['centro'][0],dfb.iloc[i]['centro'][1]))
        gdfo = gdfo.sort_values(by='dist')
        for j in range(2):  # 3 first polygons sorted by ascending distance
            XYPtj = gdfo.iloc[j]['coordpoints']
            compteur = 0
            temp = []
            for l in XYPtj:
                dist = self.distancepoint([dfb.iloc[i]['centro'], l])
                temp.append(dist)
            for d in temp:
                if d < dfb.iloc[i]['valeurseuil']:  #treshold
                    compteur += 1  #
            if compteur >= 2 and gdfo.iloc[j]['surf'] >= 1.5*dfb.iloc[i]['surf']:
                gdfo.iloc[j]['surf']-= 1.5*dfb.iloc[i]['surf']  # on enleve du potentiel à hypothese 1:1.5
                dfb.iloc[i]['geot'] = 'oui'
        m += 1
        print('avancement : '+ str(m) + '/ ' + str(fin))
    dfb.to_csv('buildingeot',sep='#', index=False)

def distancepoint(self, xy):
    "distance euclidienne,le systeme de coordonnees nest pas precise"
    if self.valeursabs(xy[0][0] - xy[1][0]) < 100000 and self.valeursabs(
            xy[0][1] - xy[1][1]) < 100000:  ##verifier limit
        d = ((xy[1][1] - xy[0][1]) ** 2 + (xy[1][0] - xy[0][0]) ** 2) ** (1 / 2)
    else:
        d = 666666
    return d

Answer 1

Especially for a problem of this size, it's worth looking for vectorized algorithms. And for many-to-many matching problems like this, numpy and scipy offer many algorithms which will outperform pandas groupby or looped options by such a significant margin that the extra effort required to manage the indices yourself is usually worth the hassle.

There are many ways to approach this, but if your goal is simply to find the nearest point using a euclidian approximation, you can't get much simpler than scipy.spatial.cKDTree . The following code finds the positional index in the second dataset (with 1M rows) for the point which is closest to each of the 100k points in the first dataset, and runs in ~20 seconds:

In [1]: import geopandas as gpd, numpy as np, scipy.spatial, pandas as pd

In [2]: # set up GeoDataFrame with 100k random points
   ...: gdf1 = gpd.GeoDataFrame(geometry=gpd.points_from_xy(
   ...:     (np.random.random(size=int(1e5)) * 360 - 180),
   ...:     (np.random.random(size=int(1e5)) * 180 - 90),
   ...: ))

In [3]: # set up GeoDataFrame with 1M random points
   ...: gdf2 = gpd.GeoDataFrame(geometry=gpd.points_from_xy(
   ...:     (np.random.random(size=int(1e6)) * 360 - 180),
   ...:     (np.random.random(size=int(1e6)) * 180 - 90),
   ...: ))


In [4]: %%time
   ...: known_xy = np.stack([gdf2.geometry.x, gdf2.geometry.y], -1)
   ...: tree = scipy.spatial.cKDTree(known_xy)
   ...: 
   ...: query_xy = np.stack([gdf1.geometry.x, gdf1.geometry.y], -1)
   ...: distances, indices = tree.query(query_xy)
   ...:
   ...:
CPU times: user 21 s, sys: 240 ms, total: 21.2 s
Wall time: 22.1 s