Dict in loop for pd.DataFrame

Question

I have many columns in my dataset & i need to change values in some of the variables. I do as below

import pandas as pd
import numpy as np
df = pd.DataFrame({'one':['a' , 'b']*5, 'two':['c' , 'd']*5, 'three':['a' , 'd']*5})

select

df1 = df[['one', 'two']]

dict

map = { 'a' : 'd', 'b' : 'c', 'c' : 'b', 'd' : 'a'}

and loop

df2=[]
for i in df1.values:
    np = [ map[x] for x in i]
    df2.append(np)

then i change columns

df['one'] = [row[0] for row in df2]
df['two'] = [row[1] for row in df2]

It works but it's very long way. How to make it shorter?

Answer 1

You can use Series.map() iterating over columns:

cols = ['one', 'two']
mapd = { 'a' : 'd', 'b' : 'c', 'c' : 'b', 'd' : 'a'}

for col in cols:
    df[col] = df[col].map(mapd).fillna(df[col])


df
Out: 
  one three two
0   d     a   b
1   c     d   a
2   d     a   b
3   c     d   a
4   d     a   b
5   c     d   a
6   d     a   b
7   c     d   a
8   d     a   b
9   c     d   a

Timings:

df = pd.DataFrame({'one':['a' , 'b']*5000000, 
                   'two':['c' , 'd']*5000000, 
                   'three':['a' , 'd']*5000000})

%%timeit
for col in cols:
    df[col].map(mapd).fillna(df[col])
1 loop, best of 3: 1.71 s per loop

%%timeit
for col in cols:
...  colSet = set(df[col].values);
...  colMap = {k:v for k,v in mapd.items() if k in colSet}
...  df.replace(to_replace={col:colMap})
1 loop, best of 3: 3.35 s per loop


%timeit df[cols].stack().map(mapd).unstack()
1 loop, best of 3: 9.18 s per loop

Answer 2

Passing whole map for col with only 'a','b' values is not efficient. At first check what values are in df col. Then map only for them, as here:

>>> cols = ['one', 'two'];
>>> map = { 'a' : 'd', 'b' : 'c', 'c' : 'b', 'd' : 'a'};

>>> for col in cols:
...  colSet = set(df[col].values);
...  colMap = {k:v for k,v in map.items() if k in colSet};
...  df.replace(to_replace={col:colMap},inplace=True);#not efficient like rly
...  
>>> df
  one three two
0   d     a   b
1   c     d   a
2   d     a   b
3   c     d   a
4   d     a   b
5   c     d   a
6   d     a   b
7   c     d   a
8   d     a   b
9   c     d   a
>>>
#OR
In [12]: %%timeit
...: for col in cols:
...:  colSet = set(df[col].values);
...:  colMap = {k:v for k,v in map.items() if k in colSet};
...:  df[col].map(colMap)
...:
...:
1 loop, best of 3: 1.93 s per loop 
#OR WHEN INPLACE
In [8]: %%timeit
   ...: for col in cols:
   ...:  colSet = set(df[col].values);
   ...:  colMap = {k:v for k,v in map.items() if k in colSet};
   ...:  df[col]=df[col].map(colMap)
   ...:
   ...:
1 loop, best of 3: 2.18 s per loop

Thats possible too:

df = pd.DataFrame({'one':['a' , 'b']*5, 'two':['c' , 'd']*5, 'three':['a' , 'd']*5})
map = { 'a' : 'd', 'b' : 'c', 'c' : 'b', 'd' : 'a'}
cols = ['one','two']

def func(s):
    if s.name in cols:
        s=s.map(map)
    return s

print df.apply(func)

Also watch for overlapping keys (ie. if You want to change in parallel lets say a to b and b to c but not like a->b->c)...

>>> cols = ['one', 'two'];
>>> map = { 'a' : 'd', 'b' : 'c', 'c' : 'b', 'd' : 'a'};
>>> mapCols = {k:map for k in cols};
>>> df.replace(to_replace=mapCols,inplace=True);
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "Q:\Miniconda3\envs\py27a\lib\site-packages\pandas\core\generic.py", line 3352, in replace
    raise ValueError("Replacement not allowed with "
ValueError: Replacement not allowed with overlapping keys and values

Answer 3

df = pd.DataFrame({'one':['a' , 'b']*5, 'two':['c' , 'd']*5, 'three':['a' , 'd']*5})
m = { 'a' : 'd', 'b' : 'c', 'c' : 'b', 'd' : 'a'}

cols = ['one', 'two']
df[cols] = df[cols].stack().map(m).unstack()
df