根据列表从熊猫系列中删除行

Question

I would like to be able to generate a list of the indices NaN values in one df and then use that list to remove the corresponding rows in a series. The goal being to have the df and the series have the same number of rows

However I keep getting stuck on the last part. If I use drop() I get not in axis error. I have tried isin() but I dont seem to get the right results

#create list of index of nan values in Garageyrblt
Index_nan_train = X_train[X_train['GarageYrBlt'].isna()].index.tolist()
# drop nan in garageyrblt X_train
X_train = X_train.drop(subset = ['GarageYrBlt'], axis = 0)
# use list to drop nan in garageyrblt Y_train
y_train = y_train.drop(['Index_nan_train'], axis = 0)

Edit: To add further details the data is from the kaggle exercise for dealing with missing values from the intermediate machine learning course.

X_train is (1168,36) dataframe with the input features and y_train is (1168,) series which represents the Sale Price

Answer 1

An easier way to do this when you're dealing with the same transformations on both, you should probably keep them together until you're done with cleaning and until you actually use the data.

x_cols = X_train.columns
y_col = y_train.name
all = pd.concat((X_train, y_train), axis=1)
all = all.dropna(subset=["GarageYrBlt"])

X_train, y_train = all.loc[:,x_cols], all.loc[:,y_col]

Alternatively use the difference index method :

nan_idx = X_train.loc[X_train["GarageYrBlt"].isna()].index
notna_idx = X_train.index.difference(nan_idx)

y_train = y_train.loc[notna_idx]

NB : Slicing on indices will mess with you if you operate on indices with groupby , reset_index and such so be aware of that problem.