熊猫从值缺失，长度不同的拆分列中创建新的DataFrame

Question

I have a bunch of categorical data that is concatenated together as a string ( string string string , etc) in each of my rows. I need to separate these categorical data into columns so that I have something that looks like this:

       var1 var2 var3
row 1:    1    1    0
row 2:    0    0    1
row 3:    0    1    1   ....

The categorical data can be generated using this:

import pandas as pd, numpy as np

np.random.seed(75)
a = [" ".join(np.unique(['var%d'%np.random.randint(5) for i in range(np.random.randint(10))])) for i in range(np.random.randint(10))]
a[0] = np.nan

b = pd.DataFrame(a, columns=['descriptor'])
print(b)

And looks like this:

                 descriptor
0                       NaN
1                 var0 var1
2       var0 var1 var2 var3
3            var0 var2 var4
4                 var0 var4
5                      var3
6  var0 var1 var2 var3 var4
7            var0 var1 var2

And I would like to construct the following

   var0  var1  var2  var3  vars4
0     0     0     0     0      0
1     1     1     0     0      0
2     1     1     1     1      0
3     1     0     1     0      1
4     1     0     0     0      1
5     0     0     0     1      0
6     1     1     1     1      1
7     1     1     1     0      0

But I'm not sure how to approach this. The DataFrame is fairly large (25,000 rows with 5000 descriptors) so the method would have to be performant. I tried using str.split(' ').tolist() , but I'm not sure how to take that and convert it to the final product.

Answer 1

First, fill the missing value with some kind of sentinel, and split the data.

b = b.fillna('_')
b['descriptor'] = b.str.split()

Then, using the approach from this answer , pivot into columns, and drop the sentinel. This won't be ultra performant because of the apply, but probably reasonable on data of your size.

s = b['descriptor'].apply(lambda x: pd.Series(1,index=x)).fillna(0).drop('_', axis=1)

In [115]: s
Out[115]: 
   var0  var1  var2  var3  var4
0     0     0     0     0     0
1     1     1     0     0     0
2     1     1     1     1     0
3     1     0     1     0     1
4     1     0     0     0     1
5     0     0     0     1     0
6     1     1     1     1     1
7     1     1     1     0     0