如何加快将列值从 pandas dataframe 传输到另一个 dataframe

Question

I have a pandas dataframe such as:

And after a complex process I want a dataframe such as:

So, I do this:

import pandas as pd

def complex_process(value):
    values=value.split(',')
    return ['results for '+x for x in values]

df=pd.DataFrame([['id1','a,b,c'],['id2','d'],['id3','e,f']],columns=['id','value'])

result_list=[]
id_list=[]
value_list=[]
for row in df.itertuples():
    results=complex_process(row.value)
    for result in results:
        result_list.append(result)
        id_list.append(row.id)
        value_list.append(row.value)
df_new=pd.DataFrame()
df_new['id']=id_list
df_new['value']=value_list
df_new['result']=result_list

This takes a long time with a large dataset. I tested the complex process and it doesn't take very long. Is there a faster way to transfer the columns?

Answer 1

Doing this operation with lists and loops is cumbersome and looping through DataFrames is computationally expensive, but pandas has lots of built-in operations so you shouldn't need to iterate through DataFrames most of the time.

Since your complex_process function is intended as a placeholder, let's apply your function to each row using .apply , and save the results in a new row called result :

df['result'] = df.value.apply(complex_process)

Your DataFrame will look like this:

>>> df
    id  value                                        results
0  id1  a,b,c  [results for a, results for b, results for c]
1  id2      d                                [results for d]
2  id3    e,f                 [results for e, results for f]

Now you can use the convenient .explode method to expand a list-like column into rows. This will duplicate the other columns and the index, so we can reset the index as well, and drop the old index.

df_new = df.explode('result').reset_index(drop=True)

Final result:

>>> df_new
    id  value         result
0  id1  a,b,c  results for a
1  id1  a,b,c  results for b
2  id1  a,b,c  results for c
3  id2      d  results for d
4  id3    e,f  results for e
5  id3    e,f  results for f