为什么我的带有 if else 子句的 for 循环运行得这么慢？

Question

TL,DR : I'm trying to understand why the below for loop is incredibly slow, taking hours to run on a dataset of 160K entries.

I have a working solution using a function and.apply(), but I want to understand why my homegrown solution is so bad. I'm obviously a huge beginner with Python:

popular_or_not = []
counter = 0
for id in df['id']:
    if df['popularity'][df['id'] == id].values == 0:
        popular_or_not.append(0)
    else:
        popular_or_not.append(1)
    counter += 1

df['popular_or_not'] = popular_or_not
df

In more detail:

I'm currently learning Python for data science, and I'm looking at this dataset on Kaggle: https://www.kaggle.com/yamaerenay/spotify-dataset-19212020-160k-tracks

I'm interesting in predicting/modelling the popularity score. It is not normally distributed:

plt.bar(df['popularity'].value_counts().index, df['popularity'].value_counts().values)

I would like to add a column, to say whether a track is popular or not, with popular tracks being those that get a score of 5 and above and unpopular being the others.

I have tried the following solution, but it runs incredibly slowly, and I'm not sure why. It runs fine on a very small subset, but would take a few hours to run on the full dataset:

popular_or_not = []
counter = 0
for id in df['id']:
    if df['popularity'][df['id'] == id].values == 0:
        popular_or_not.append(0)
    else:
        popular_or_not.append(1)
    counter += 1

df['popular_or_not'] = popular_or_not
df

This alternative solution works fine:

def check_popularity(score):
    if score > 5:
        return 1
    else:
        #pdb.set_trace()
        return 0
df['popularity'].apply(check_popularity).value_counts()
df['popular_or_not'] = df['popularity'].apply(check_popularity)

I think understanding why my first solution doesn't work might be an important part of my Python learning.

Answer 1

Thanks everyone for your comments. I'm going to summarize them below as an answer to my question, but please feel free to jump in if anything is incorrect:

The reason my initial for loop was so slow is that I was checking df['id'] == id 160k times. This is typically a very slow operation.

For this type of operation, instead of iterating over a pandas dataframe thousands of times, it's always a good idea to think of applying vectorization - a bunch of tools and methods to process a whole column in a single instruction at C speed. This is what I did with the following code:

def check_popularity(score):
    if score > 5:
        return 1
    else:
        #pdb.set_trace()
        return 0
df['popularity'].apply(check_popularity).value_counts()
df['popular_or_not'] = df['popularity'].apply(check_popularity)

By using.apply and a pre-defined function. I get the same result, but in seconds instead of in hours.