如何解决 pandas.get_dummies 中的问题

Question

I'm preprocessing my dataset with pd.get_dummies, but the result is not what I need.

Is it correct to use pd.get_dummies()? Or any approaches I can try?

import pandas as pd
rawdataset=[['apple','banana','carrot','daikon','egg'],
           ['apple','banana'],
           ['apple','banana','carrot'],
           ['daikon','egg','fennel'],
           ['apple','banana','daikon']]
dataset=pd.DataFrame(data=rawdataset)
print(pd.get_dummies(dataset))

I expect it looks like this:

   apple banana carrot daikon egg fennel 

0   1      1      1     1     1    0
1   1      1      0     0     0    0
........  

not like this:

   0_apple  0_daikon  1_banana  1_egg  2_carrot  2_daikon  2_fennel  

0    1         0          1       0       1         0           0
1    1         0          1       0       0         0           0
....

Answer 1

Different ways to skin a cat.

---

pd.get_dummies and max

pd.get_dummies(dataset, prefix="", prefix_sep="").max(level=0, axis=1)

   apple  daikon  banana  egg  carrot  fennel
0      1       1       1    1       1       0
1      1       0       1    0       0       0
2      1       0       1    0       1       0
3      0       1       0    1       0       1
4      1       1       1    0       0       0

---

stack , str.get_dummies , and sum / max :

df.stack().str.get_dummies().sum(level=0)

   apple  banana  carrot  daikon  egg  fennel
0      1       1       1       1    1       0
1      1       1       0       0    0       0
2      1       1       1       0    0       0
3      0       0       0       1    1       1
4      1       1       0       1    0       0

---

stack and crosstab

u =  df.stack()
pd.crosstab(u.index.get_level_values(0), u)

col_0  apple  banana  carrot  daikon  egg  fennel
row_0                                            
0          1       1       1       1    1       0
1          1       1       0       0    0       0
2          1       1       1       0    0       0
3          0       0       0       1    1       1
4          1       1       0       1    0       0

Answer 2

Here you go:

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

rawdataset=[['apple','banana','carrot','daikon','egg'],
            ['apple','banana'],
            ['apple','banana','carrot'],
            ['daikon','egg','fennel'],
            ['apple','banana','daikon']]


def dummy(doc):
    return doc

count_vec = CountVectorizer(tokenizer=dummy, preprocessor=dummy)

count_vec.fit(rawdataset)

X = count_vec.transform(rawdataset).todense()

pd.DataFrame(X, columns=count_vec.get_feature_names())

Results in:

   apple  banana  carrot  daikon  egg  fennel
0      1       1       1       1    1       0
1      1       1       0       0    0       0
2      1       1       1       0    0       0
3      0       0       0       1    1       1
4      1       1       0       1    0       0

The added benefit here is that you can also apply it to unseen data as pd.get_dummies cannot convert other unseen test data in the same way.

Try:

unseen_raw_data = [["test"]]
feature_names = count_vec.get_feature_names()
unseen_data = count_vec.transform(unseen_raw_data).todense()
pd.DataFrame(unseen_data, columns=feature_names)

yields:

   apple  banana  carrot  daikon  egg  fennel
0      0       0       0       0    0       0

which is the correct output