如何按列分组并将其他列的值作为列表返回到pandas中？

Question

I am facing trouble clubbing the values of a column together and retaining the respective values of other columns. I would like to do something similar to this: grouping rows in list in pandas groupby

But instead, I want the list/dictionary (preferably latter) to contain values of multiple columns. Example for this dataframe:

df:

Col1   Col2   Col3
A      xyz     1
A      pqr     2
B      xyz     2
B      pqr     3
B      lmn     1
C      pqr     2

I want something like-

A {'xyz':1, 'pqr': 2}
B {'xyz':2, 'pqr': 3, 'lmn': 1}
C {'pqr':2}

I tried doing

df.groupby('Col1')[['Col2', 'Col3']].apply(list) 

which is a variant of the solution mentioned in the linked post, but isn't giving me the result I need.

From that point on, I would also like to transform it into a dataframe of the form:

  xyz  pqr  lmn
A  1    2    NaN
B  2    3    1
C  NaN  2    NaN

Answer 1

What you want in the end is a pivot table.

df.pivot_table(index='Col1',columns='Col2',values='Col3')

look up the documentation for more options.

Answer 2

Use pivot or unstack :

df = df.pivot(index='Col1',columns='Col2',values='Col3')
print (df)
Col2  lmn  pqr  xyz
Col1               
A     NaN  2.0  1.0
B     1.0  3.0  2.0
C     NaN  2.0  NaN

---

df = df.set_index(['Col1','Col2'])['Col3'].unstack()
print (df)

Col2  lmn  pqr  xyz
Col1               
A     NaN  2.0  1.0
B     1.0  3.0  2.0
C     NaN  2.0  NaN

but if:

ValueError: Index contains duplicate entries, cannot reshape

it means duplicates, need pivot_table or aggregate with groupby by mean (can be changed to sum , median ), and last reshape by unstack :

print (df)
  Col1 Col2  Col3
0    A  xyz     1 <-same A, xyz
1    A  xyz     5 <-same A, xyz
2    A  pqr     2
3    B  xyz     2
4    B  pqr     3
5    B  lmn     1
6    C  pqr     2

df = df.groupby(['Col1','Col2'])['Col3'].mean().unstack()
print (df)
Col2  lmn  pqr  xyz
Col1               
A     NaN  2.0  3.0 (1+5)/2 = 3
B     1.0  3.0  2.0
C     NaN  2.0  NaN

EDIT:

For check all duplicates by Col1 and Col2 :

print (df[df.duplicated(subset=['Col1','Col2'], keep=False)])
  Col1 Col2  Col3
0    A  xyz     1
1    A  xyz     5

EDIT1:

If need only first row if duplicates:

df = df.groupby(['Col1','Col2'])['Col3'].first().unstack()
print (df)
Col2  lmn  pqr  xyz
Col1               
A     NaN  2.0  1.0
B     1.0  3.0  2.0
C     NaN  2.0  NaN

Or better first remove duplicates by drop_duplicates and then use first or second solution:

df = df.drop_duplicates(subset=['Col1','Col2'])
df = df.pivot(index='Col1',columns='Col2',values='Col3')
print (df)
Col2  lmn  pqr  xyz
Col1               
A     NaN  2.0  1.0
B     1.0  3.0  2.0
C     NaN  2.0  NaN

Answer 3

Neither one of these are pandas only solutions. I provided them because I find exploring alternatives fun. The bincount base solution is very fast but less transparent.

Creative Solution 1
collections.defaultdict and dictionary comprehension

from collections import defaultdict

d = defaultdict(dict)
[d[c2].setdefault(c1, c3) for i, c1, c2, c3 in df.itertuples()];
pd.DataFrame(d)

   lmn  pqr  xyz
A  NaN    2  1.0
B  1.0    3  2.0
C  NaN    2  NaN

---

Creative Solution 2
pd.factorize and np.bincount

f1, u1 = pd.factorize(df.Col1.values)
f2, u2 = pd.factorize(df.Col2.values)
w = df.Col3.values

n, m = u1.size, u2.size

v = np.bincount(f1 * n + f2, w, n * m).reshape(n, m)
pd.DataFrame(np.ma.array(v, mask=v == 0), u1, u2)

   lmn  pqr  xyz
A  NaN    2  1.0
B  1.0    3  2.0
C  NaN    2  NaN

---

Timing

%timeit df.pivot(index='Col1',columns='Col2',values='Col3')
%timeit df.set_index(['Col1','Col2'])['Col3'].unstack()
%timeit df.groupby(['Col1','Col2'])['Col3'].mean().unstack()
%timeit df.pivot_table(index='Col1',columns='Col2',values='Col3')

%%timeit
d = defaultdict(dict)
[d[c2].setdefault(c1, c3) for i, c1, c2, c3 in df.itertuples()];
pd.DataFrame(d)

%%timeit
f1, u1 = pd.factorize(df.Col1.values)
f2, u2 = pd.factorize(df.Col2.values)
w = df.Col3.values

n, m = u1.size, u2.size

v = np.bincount(f1 * n + f2, w, n * m).reshape(n, m)
pd.DataFrame(np.ma.array(v, mask=v == 0), u1, u2)

small data

1000 loops, best of 3: 1.11 ms per loop
1000 loops, best of 3: 1.67 ms per loop
1000 loops, best of 3: 1.51 ms per loop
100 loops, best of 3: 4.17 ms per loop

1000 loops, best of 3: 1.18 ms per loop

1000 loops, best of 3: 420 µs per loop

medium data

from string import ascii_letters
l = list(ascii_letters)
df = pd.DataFrame(dict(
        Col1=np.random.choice(l, 10000),
        Col2=np.random.choice(l, 10000),
        Col3=np.random.randint(10, size=10000)
    )).drop_duplicates(['Col1', 'Col2'])

1000 loops, best of 3: 1.75 ms per loop
100 loops, best of 3: 2.17 ms per loop
100 loops, best of 3: 2.2 ms per loop
100 loops, best of 3: 4.89 ms per loop

100 loops, best of 3: 5.6 ms per loop

1000 loops, best of 3: 549 µs per loop