使用 sklearn.preprocessing.LabelEncoder() 使用 Python 编码多个分类数据在 2D 数组输入上需要太多处理时间

Question

Consider for some reason I am trying to encode a feature. Let's say my feature name is title . For the title feature, for one record I might have different words: title = 'Apple', 'Jobs'. Let me illustrate:

ID      title  
0  ['Apple', 'Jobs']  
1  ['Wozniak']
2  ['Apple', 'Wozniak']
3  ['Jobs', 'Wozniak']  

As you could see my unique values are:

unique = ['Apple','Jobs','Wozniak']

And previously I was using label encoder as:

from sklearn.preprocessing import LabelEncoder

le.fit(unique)
for i in df['title'].index:
    df['title'][i] = le.transform(df['title'][i])

And I used to get something like:

ID      title  
0  [782, 256]  
1  [331]
2  [782, 331]
3  [256, 331] 

which was exactly what I wanted; yet, this takes too much time because I have too many values to iterate and encode. Thus, I am looking for an algorithm that is smarter and preferably with a lower time complexity or smaller running time.

Answer 1

Later, I have discovered that first partitioning the elements of title up to 5 columns and applying label encoding solves my problem. I am sharing a sample of solution:

def process(self):
    for i in range(self._MULHOTBEGIN,self._MULHOTEND):
        print("\tWorking on",self.df.columns[i])
        self._colnames.append(self.df.columns[i])
        self.df = pd.merge(self.df,self._processSubRoutine(i), left_index=True, right_index=True)
    print("\tCollecting garbage.")
    self._collectGarbage()

def _processSubRoutine(self, colindex):
    result = list()
    for i in range(self._len):
        truncated = self.df.iloc[i,colindex][:self._MAXLEN]
        padded = list(truncated) + ['0']*(self._MAXLEN-len(truncated))
        result.append(padded)
    colnames = self._createColNames(colindex)
    print("\t",colnames)
    return pd.DataFrame(result,columns=colnames, index=self.df.index)

These processes are called inside the preprocessing file within

M = MulHotProcessor(reorderedSource,mulhotBeginIndex2, mulhotEndIndex2, max_len_per_slot_2)
M.process()
sourceProc = M.getDataFrame()

entvals = pd.concat([sourceProc['title_0'],sourceProc['title_1'], \
              sourceProc['title_2'],sourceProc['title_3'], \
              sourceProc['title_4']]).unique()

Later on we finally apply label encoding by fitting the unique values

le.fit(<unique_values>) # whatever you name your unique values

sourceProc['title_0']=le.transform(sourceProc['title_0'])
sourceProc['title_1']=le.transform(sourceProc['title_1'])
sourceProc['title_2']=le.transform(sourceProc['title_2'])
...

At the end you will have in very short time prepared the transformation from first DF to second DF in the question.