在 pandas dataframe 中迭代超过 7000 万行的最快方法

Question

I have two pandas dataframes bookmarks and ratings where columns are respectively:

• id_profile , id_item , time_watched
• id_profile , id_item , score

I would like to find score for each couple (profile,item) in the ratings dataframe (set to 0 if does not exist). The problem is the bookmarks dataframe has 73 million rows and it takes so much time (after 15 min the code continues to run). I suppose there is a better way to do it.

Here is my code:

def find_rating(val):
  res = ratings.loc[(ratings['id_profile'] == val[0]) & (ratings['id_asset'] == val[1])]
  if res.empty :
    return 0
  return res['score'].values[0]

arr = bookmarks[['id_profile','id_asset']].values
rates = [find_rating(i) for i in arr]

I work on collab.

Do you think I can improve speed of the execution?

Answer 1

Just some thoughts. I have not tried such large data in Pandas.

In pandas, the data is index on row as well as columns. So if you have 1 million rows with 5 columns ==> we have indexed 5 million records.

For performance boost,

1. Check if you can use Sparse* data structures - https://pandas.pydata.org/pandas-docs/stable/user_guide/sparse.html

2. Filter out the unnecessary data as much feasible.

3. If you can, try stick to using only numpy. As this reduces few features, we also we loose some drag. Worth exploring.

4. Use some distributed multithreaded/multiprocessing tools like Dask\Ray. If you have 4 cores -> 4 parallel jobs => 25% faster