在for循环python中编码和保存多个数组

Question

I am preprocessing data for a deep neural network and I have variables that I need to one hot encode. So far, this is what I am doing and it is working fine. However, I was wondering if I could implement this in a for loop as that may be more efficient?

# Only Educational Establishment Type
X6 = X.drop(['Sex', 'Applicant Domicile (High Level)', 'Applicant Domicile (Low Level)', 'Age Band (5 Levels)', 'POLAR3 Quintile', 'Subject Group (Detailed Level)', 'Subject Group (Summary Level)'], axis=1)
X6 = onehotencoder.fit_transform((X6).apply(encoder.fit_transform)).toarray()
# print(X6.head)

# Only Subject Group (Detailed Level)
X7 = X.drop(['Sex', 'Applicant Domicile (High Level)', 'Applicant Domicile (Low Level)', 'Age Band (5 Levels)', 'POLAR3 Quintile', 'Educational Establishment Type', 'Subject Group (Summary Level)'], axis=1)
X7 = onehotencoder.fit_transform((X7).apply(encoder.fit_transform)).toarray()
# print(X7.head)

# Only Subject Group (Summary Level)
X8 = X.drop(['Sex', 'Applicant Domicile (High Level)', 'Applicant Domicile (Low Level)', 'Age Band (5 Levels)', 'POLAR3 Quintile', 'Educational Establishment Type', 'Subject Group (Detailed Level)'], axis=1)
X8 = onehotencoder.fit_transform((X8).apply(encoder.fit_transform)).toarray()

I also need to save the encoded array in a npy format to recall later. I attempted to implement all of this in a for loop as follows; however, it does not save a file for each array as desired and it does not actually update the existing dataframes into one hot encoded arrarys.

all_x = [X, X1, X2, X3, X4, X5, X6, X7, X8]

idx = 0
for num_x in all_x: 
   encoder=LabelEncoder()
   (num_x) = (num_x).apply(encoder.fit_transform)   
   onehotencoder = OneHotEncoder(categorical_features='all')
   (num_x) = onehotencoder.fit_transform(num_x).toarray()
   np.save('X%d' % idx, num_x)
   idx+=1
   print(num_x)

Answer 1

You're actually not updating the existing dataframes, in either version; you're just creating new dataframes.

The difference is that in the repetitive code, you're rebinding each of the variables X , X1 , etc. to each of those new dataframes, while in your loop, you're rebinding num_x over and over again to each new dataframe, so at the end, only the last one is stored anywhere.

Ideally, you probably want to rewrite your code to not have those separate variables in the first place, and only have a list of 9 dataframes—or maybe a dict of 9 dataframes keyed by names X , X1 , etc., if the names really are meaningful.

But if that's too big of a change, you can do something like this:

all_x = [X, X1, X2, X3, X4, X5, X6, X7, X8]
idx = 0
for num_x in all_x:
   encoder=LabelEncoder()
   (num_x) = (num_x).apply(encoder.fit_transform)   
   onehotencoder = OneHotEncoder(categorical_features='all')
   # store the new dataframe back into the list
   all_x[idx] = num_x = onehotencoder.fit_transform(num_x).toarray()
   np.save('X%d' % idx, num_x)
   idx+=1
   print(num_x)
# store the list of new dataframes back into the original variable names
X, X1, X2, X3, X4, X5, X6, X7, X8 = all_x

There are ways you can make this cleaner (eg, using for idx, num_x in enumerate(all_x): ), but I stuck as closely as possible to your existing code.