如何比较csv文件的三列并使用python确定丢失的数据？

Question

I have a CSV file (screenshot attached)

There I have three different columns and the first one, "Set of Ids known for transformers" is the master column. I need to compare the other two columns with the master column and filter out the missing values in the remaining two columns.

Can anyone please tell me how to do this thing in python with the pandas library?

Thanks in advance.

Answer 1

You can remove any row missing both nonmaster columns using a condition to select rows:

import pandas as pd
df = pd.read_excel('soxl.xlsx')
df=df[df['IDs of phase known'].notnull() | df['Ids of distance known'].notnull()]

If you wanted to remove any row with any missing data you can use the builtin :

df = df.dropna()

which removes any row with missing values (which otherwise get imported as Nan values).

If you want to replace the 2nd col. with master column values when they are missing, you can do

df = df.where((pd.notnull(df)), None)

to replace the NaN's with None (useful in the next step) and then

df['IDs of phase known']= df.apply(lambda r:r['IDs of phase known'] if r['IDs of phase known'] else r['Set of Ids known for transformers'],axis=1)

And of course you can do the same with the 3rd col. If you want to replace only in the case where both col2 and 3 values are missing, you can do similarly, but check both columns for None:

df['IDs of phase known']= df.apply(lambda r:r['Set of Ids known for transformers'] if not (r['IDs of phase known'] or r['Ids of distance known']) else r['IDs of phase known'] ,axis=1 )

You can also do this, actually a bit easier, with numpy:

df['IDs of phase known'] = np.where(np.isnan(df['IDs of phase known']), df['Set of Ids known for transformers'], df['IDs of phase known'])

If you want to just replace missing values with master column vals, or

df['IDs of phase known'] = np.where( (np.isnan(df['IDs of phase known']) & (np.isnan(df['Ids of distance known']))),df['Set of Ids known for transformers'],df['IDs of phase known'])

If you only want to replace in cases where both cols are missing.