在Pandas / Python中使用可变大小的行更新数据框

Question

I have imported an excel sheet into a dataframe in Pandas. The blank values were replaced by 'NA's. What I want to do is, for each of the row values, replace them based on indices of a dictionary or dataframe.

df1 = pd.DataFrame(
    {'c1':['a','a','b','b'], 'c2':['1','2','1','3'], 'c3':['2','NA','3','NA']},index=['first','second','third','last'])

>>> df1
       c1 c2  c3
first  a  1    2
second a  2    NA
third  b  1    3
last   b  3    NA

and I want to replace the values in each row according to the indices of another dataframe (or dict).

df2=pd.DataFrame(
    {'val':['v1','v2','v3']},index=['1','2','3'])

>>> df2
   val
1  v1  
2  v2 
3  v3 

Such that the output becomes

>>> out
       c1 c2  c3
first  a  v1  v2
second a  v2  NA
third  b  v1  v3
last   b  v3  NA

How would you do this through Pandas and/or Python? One way to do it would be to search row by row, but maybe there is an easier way?

Edit: Importantly, performance becomes an issue in my real case since I am dealing with a 'df1' whose size is 4653 rows × 1984 columns .

Thank you in advance

Answer 1

One way would be stack + replace + unstack combo:

df1.stack().replace(df2.val).unstack()

Answer 2

Original answer

s = df1.squeeze()
df2.replace(s)

replace is very, very slow. For a larger data set like you have check the following example which is done over 30 million values (more than your 10 million values) in about 20 seconds. The lookup Series contains 900k values from 0 to 1 million.

'map' is much, much faster. The only issue with map is that it replaces a value not found with missing so you will have to use fillna with the original DataFrame to replace those missing values.

n = 10000000
df = pd.DataFrame({'c1':np.random.choice(list('abcdefghijkl'), n),
                 'c2':np.random.randint(0, 1000000, n),
                 'c3':np.random.randint(0, 1000000, n)})

s = pd.Series(index=np.random.choice(np.arange(1000000), 900000, replace=False), 
              data=np.random.choice(list('adsfjhqwoeriouzxvmn'), 900000, replace=True))

df.stack().map(s).unstack().fillna(df)

You can also do this which is running faster on my data but your data is very wide so it might be slower

df.apply(lambda x: x.map(s)).fillna(df)

And on a DataFrame similar to yours, I am getting 6s to complete.

df = pd.DataFrame(np.random.randint(0, 1000000, (5000, 2000)))
df.stack().map(s).unstack().fillna(df)