Hyperparameter tuning in keras using nested k-fold cross-validation

Question

RandomizedSearchcv accepts only a one-dimensional target variable, but for this binary classification I need to convert y_train and y_test to one-hot variable to keras. I got error 'Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead.' Could anyone give me some tips? Many thanks!

def create_baseline():
  model = Sequential()
  model.add(Reshape((TIME_PERIODS, num_sensors), input_shape=(input_shape,)))
  model.add(Conv1D(100, 6, activation='relu', input_shape=(TIME_PERIODS, num_sensors)))
  #model.add(BatchNormalization())
  model.add(Dropout(0.5))
  model.add(MaxPooling1D(3))
  model.add(Conv1D(100, 6, activation='relu'))
  model.add(Dropout(0.5))
  model.add(MaxPooling1D(3))
  # LSTM
  model.add(LSTM(64,return_sequences=True))
  model.add(Dropout(0.5))
  model.add(LSTM(32,return_sequences=True))
  model.add(Dropout(0.5))
  model.add(Dense(128, activation="sigmoid", kernel_initializer="uniform"))
  model.add(Dropout(0.5))
  model.add(GlobalAveragePooling1D())
  model.add(Flatten())
  model.add(Dense(num_classes, activation='softmax'))

  model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
  return model


from sklearn.model_selection import StratifiedKFold,KFold
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
seed=42
#y_train = np_utils.to_categorical(y_train, num_classes)
estimator = KerasClassifier(build_fn=create_baseline, epochs=30, batch_size=800, verbose=1)



# Nested k-fold cross-validation (Subject_dependent)

from sklearn.model_selection import GridSearchCV,cross_val_score, StratifiedKFold,RandomizedSearchCV
#train/validation/test=0.8/0.2/0.2
inner_cv = StratifiedKFold(n_splits = 4,shuffle=True,random_state=42)
outer_cv = StratifiedKFold(n_splits = 5,shuffle=True,random_state=42)

accuracy=[]
p_grid=[]
estimators=[]
#p_grid={'batch_size':[400,800]}

from sklearn.preprocessing import LabelEncoder

#def get_new_labels(y):
    #y = LabelEncoder().fit_transform([''.join(str(l)) for l in y])
    #return y
#y = get_new_labels(y)

for train_index, test_index in outer_cv.split(x,y):
    print('Train Index:',train_index,'\n')
    print('Test Index:',test_index)
    x_train, x_test = x[train_index], x[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    y_train = np_utils.to_categorical(y_train, num_classes)
    y_test = np_utils.to_categorical(y_test, num_classes) 
    grid = RandomizedSearchCV(estimator=estimator,
                                param_distributions=p_grid,
                                cv=inner_cv,                            
                                refit='roc_auc_scorer',
                                return_train_score=True,
                                verbose=1,n_jobs=-1,n_iter=20)
    grid.fit(x_train, y_train)
    estimators.append(grid.best_estimator_)
    prediction = grid.predict(x_test)
    accuracy.append(grid.score(x_test,y_test))
    print('Accuracy:{}'.format(accuracy))

Answer 1

In binary classification it's either a dog, or not a dog, and your encoded labels would just be 1's or 0's:

[[0] <- single row label
 [1] <- single row label
 [0]]

In multiclass classification it's either a dog, cat, or a bird, and not more than one ie they are mutually exclusive So your encoded labels look like:

[[0,0,1] <-- a single rows encoded label
  [1,0,0] <-- another rows encoded label
  [0,1,0]]

Multilabel classification is different, it accepts label sets which are not mutually exclusive ie it can be a building, as well as a house, as well as an office ie:

[[1,1,1]
 [0,0,1]
 [1,0,1]]

The problem here is that it looks like you're passing multilabel labels to your classifier - you should double check your labels and make sure that there is only a 1 or a 0 for each row of training data if that is what you need.

Using to_categorical for binary classification is fine, however you might want to double check that num_classes=2 for binary classification.

Also, if it is a binary classification problem, your final Dense layer activation needs to be 'sigmoid' not 'softmax'. See here for notes.