将具有分类值的熊猫数据框转换为二进制值

Question

I am trying to convert categorical data into binary to be able to classify with an algorithm like logistic regression . I thought of using OneHotEncoder from 'sklearn.preprocessing' module but the problem is the dataframe entries are A, B pairs of arrays with different lengths, each row has pair of same-length arrays not equal to array lengths in other rows. OneHotEncoder does not accept dataframe like mine

In [34]: data.index

Out[34]: Index([train1, train2, train3, ..., train7829, train7830, train7831], dtype=object)

In [35]:  data.columns

Out[35]:  Index([A, B], dtype=object)

SampleID                      A                                B
train1      [2092.0, 1143.0, 390.0, ...]          [5651.0, 4449.0, 4012.0...]
train2      [3158.0, 3158.0, 3684.0, 3684.0....]  [2.0, 4.0, 2.0, 1.0...]
train3      [1699.0, 1808.0 ,...]                 [0.0, 1.0...]

So, I want to highlight again that each A and B pair has the same length but the length is variable across different pairs. Dataframe contains numerical, categorical and binary values. I have another csv file with the information about every entry type. I read the file filter out categorical entries in both columns like this:

info=data_io.read_train_info()
col1=info.columns[0]
col2=info.columns[1]
info=info[(info[col1]=='Categorical')&(info[col2]=='Categorical')]

Then I use info.index to filter my training dataframe

filtered = data.loc[info.index]

Than I wrote an utility function to change dimensions of each array so that I can encode them later

def setDim(df):
    for item in x[x.columns[0]].index:
        df[df.columns[0]][item].shape=(1,df[df.columns[0]][item].shape[0])
        df[df.columns[1]][item].shape=(1,df[df.columns[1]][item].shape[0])

setDim(filtered)

Then I thought to combine each pair of arrays into 2-row matrix so that I can pass it to encoder then to separate them again after encoding, like this:

import numpy as np
from sklearn.preprocessing import OneHotEncoder

def makeSparse(df):
   enc = OneHotEncoder()
   for i in df.index:
     cd=np.append(df['A'][i],df['B'][i],axis=0)
     a=enc.fit_transform(cd)
     df['A'][i] = a[0,:]
     df['B'][i] = a[1,:]

makeSparse(filtered)

After all these steps get a sparse dataframe. My questions are:

1. is this a right way to encode this dataframe?(I highly doubt it)
2. if no, then what alternatives do you offer?
   Thanks a lot for your time helping me.

Answer 1

This is a nice way to transform your data to a better repr to deal with; uses some neat apply tricks

In [72]: df
Out[72]: 
                               A                  B
train1         [2092, 1143, 390]  [5651, 449, 4012]
train2  [3158, 3158, 3684, 3684]       [2, 4, 2, 1]
train3              [1699, 1808]             [0, 1]

In [73]: concat(dict([ (x[0],x[1].apply(lambda y: Series(y))) for x in df.iterrows() ]))
Out[73]: 
             0     1     2     3
train1 A  2092  1143   390   NaN
       B  5651   449  4012   NaN
train2 A  3158  3158  3684  3684
       B     2     4     2     1
train3 A  1699  1808   NaN   NaN
       B     0     1   NaN   NaN