如何过滤此python数据帧

Question

Greeting I try to get the smallest sizes dataframe that got valid row

import pandas as pd
import random

columns = ['x0','y0']
df_ = pd.DataFrame(index=range(0,30), columns=columns)
df_ = df_.fillna(0)


columns1 = ['x1','y1']

df = pd.DataFrame(index=range(0,11), columns=columns1)



for index, row in df.iterrows():
   df.loc[index, "x1"] = random.randint(1, 100)
   df.loc[index, "y1"] = random.randint(1, 100)

df_ = df_.combine_first(df)    

df = pd.DataFrame(index=range(0,17), columns=columns1)



for index, row in df.iterrows():
    df.loc[index, "x2"] = random.randint(1, 100)
    df.loc[index, "y2"] = random.randint(1, 100)

df_ = df_.combine_first(df)    

From the example the dataframe should output rows from 0 to 10 and the rest got filter out. I think of keep a counter to keep track of the min row or using pandasql or if there is a trick to get this info from the dataframe the size of dataframe

Actually I will be appending 500+ files with various size to append and use it to do some analysis. So perf is a consideration.

-student of python

Answer 1

If you want to drop the rows which have NaNs use dropna (here, this is the first ten rows):

In [11]: df_.dropna()
Out[11]:
    x0  x1  x2  y0  y1  y2
0    0  49  58   0  68   2
1    0   2  37   0  19  71
2    0  26  95   0  12  17
3    0  87   5   0  70  69
4    0  84  77   0  70  92
5    0  71  98   0  22   5
6    0  28  95   0  70  15
7    0  31  19   0  24  31
8    0   9  37   0  55  29
9    0  30  53   0  15  45
10   0   8  61   0  74  41

---

However a cleaner, more efficient, and faster way to do this entire process is to update just those first rows (I'm assuming the random integer stuff is just you generating some example dataframes).

Let's store your DataFrames in a list:

In [21]: df1 = pd.DataFrame([[1, 2], [np.nan, 4]], columns=['a', 'b'])

In [22]: df2 = pd.DataFrame([[1, 2], [5, 6], [7, 8]], columns=['a', 'c'])

In [23]: dfs = [df1, df2]

Take the minimum length:

In [24]: m = min(len(df) for df in dfs)

First create an empty DataFrame with the desired rows and columns:

In [25]: columns = reduce(lambda x, y: y.columns.union(x), dfs, [])

In [26]: res = pd.DataFrame(index=np.arange(m), columns=columns)

To do this efficiently we're going to update, and making these changes inplace - on just this DataFrame*:

In [27]: for df in dfs:
             res.update(df)

In [28]: res
Out[28]:
   a  b  c
0  1  2  2
1  5  4  6

*If we didn't do this, or were using combine_first or similar, we'd most likely have lots of copying (new DataFrames being created), which will slow things down.

Note: combine_first doesn't offer an inplace flag... you could use combine but this is also more complicated (as well as less efficient). It's also quite straightforward to use where (and manually update), which IIRC is what combine does under the hood.