Pandas not dropping rows and columns that meet criteria

Question

I am trying to make a regression model in order to predict ratings (1-5) based on words that appear (the regression doesn't have to perform well per se, it's more about the methodology applied). I created a term frequency matrix with this code:

bow = df.Review2.str.split().apply(pd.Series.value_counts)

which look like this:

I am now interested in deleting columns (words) that rarely appear throughout the reviews. Moreover, I want to iterate through only the reviews (rows) that have a Rating value which is not NaN .

here is my attempt:

# Delete row if Rating less than 1
for index, row in df.iterrows():
    if (df.Rating[index] < 1):
        bow.drop(bow.index[index], axis=0, inplace = True)

# Delete column if word occurs less than 50 times
sum1 = bow.sum(axis=0)       
cntr = 0
for i in sum1:
    if (i < 50):
        bow.drop(bow.index[cntr], axis=1, inplace = True)
    cntr += 1

This doesn't seem to do the work as it leaves words that occur only once.

---

EDIT:

This is my sparse dataframe containing occurrences of words. Col -> words; Rows -> sentences (item's reviews) (I have 1.5k items, thus 1.5k rows)

     hi this are just some random words  I  don t      ...  zing  zingy zingzang    
0   1.0 NaN  1.0 1.0  1.0   NaN   NaN   NaN NaN NaN    ...  NaN    NaN    NaN   
1   NaN NaN  NaN NaN  NaN   NaN   NaN   NaN NaN NaN    ...  NaN    NaN    NaN       
2   NaN NaN  NaN NaN  NaN   NaN   NaN   NaN NaN NaN    ...  NaN    NaN    NaN   
3   NaN NaN  NaN NaN  NaN   NaN   NaN   NaN NaN NaN    ...  NaN    NaN    NaN   
4   NaN NaN  NaN NaN  NaN   NaN   NaN   NaN NaN 1.0    ...  NaN    NaN    NaN   

Rating is a single column of my original dataframe containing integers in the [1,5] range or NaN

Answer 1

You can use vectorised operations instead of manual iteration:

# filter out rows where Rating < 1
bow = bow[~(bow['Rating'] < 1)]

# filter out columns where sum < 50
bow = bow.loc[:, ~(bow.sum(0) < 50)]

Or simultaneously:

# filter rows and columns with Boolean series
bow = bow.loc[~(bow['Rating'] < 1), ~(bow.sum(0) < 50)]

Answer 2

I made this working toy example:

import pandas as pd
import numpy  as np

# Create a toy daframe
df = pd.DataFrame(np.arange(12).reshape(3,4),columns=['A', 'B', 'C', 'D'])

print(df)
#   A   B  C  D
#-------------
# 0  0  1  2  3
# 1  4  5  6  7
# 2  8  9 10 11

# Sum all the values for each column
column_sum = df.sum(axis=0)
print(column_sum)
# A    12
# B    15
# C    18
# D    21

# Iterate over Columns name and sum value
for key,value in zip(df.keys(),sum1):
    if(value < 16):
        df.drop(columns=key, axis=1, inplace = True)

print(df)

#    C  D
# 0  2  3
# 1  6  7
# 2 10 11

so I guess that if you change your code to:

for key,value in zip(df.keys(),sum1):
    if(value < 50):
        bow.drop(columns=key, axis=1, inplace = True)

it should get the job done.