Param Tuning with Keras and Hyperas

Question

I have been working with a Python library called Hyperas which is a hyperopt/keras wrapper for tuning parameters in a Keras model. My question is about the output from Hyperas.

I have read the documentation and source code, but cannot seem to figure out what the output means or how to interpret. It prints the following line after it is done optimizing:

{'batch_size': 3, 'optimizer': 1, 'l2': 0.7446290506725413, 'output_dim': 3, 'output_dim_1': 0, 'l2_1': 0.12090219120950985}

Why are there two dictionary values for output_dim, even though there is only one output_dim param in my code? How do I interpret everything elese?

def model(X_train, X_test, y_train, y_test, max_features, maxlen, class_weight_dict):
        model = Sequential()
        model.add(Embedding(max_features, output_dim = {{choice([32,64,128,256,512])}}, input_length=maxlen))
        model.add(LSTM({{choice([32,64,128,256,512])}},W_regularizer=l2({{uniform(0, 1)}})))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense(138))
        model.add(Activation('softmax'))

        model.compile(loss='categorical_crossentropy',
                      optimizer={{choice(['rmsprop', 'adam', 'sgd'])}},
                      metrics=['accuracy'])

        early_stopping = EarlyStopping(monitor='val_loss', patience=4)
        checkpointer = ModelCheckpoint(filepath='keras_weights.hdf5',
                                       verbose=1,
                                       save_best_only=True)

        model.fit(X_train, y_train,
                  batch_size={{choice([32,16,64,128,256,512])}},
                  validation_data=(X_test, y_test),
                  nb_epoch=100,
                  class_weight=class_weight_dict,
                  callbacks=[early_stopping, checkpointer])

        score, acc = model.evaluate(X_test, y_test)
        print('Test score:', score)
        print('Test accuracy:', acc)
        return {'loss': -acc, 'status': STATUS_OK, 'model': model}

    if __name__ == '__main__':
        best_run, best_model = optim.minimize(model=model,
                                              data=data,
                                              algo=tpe.suggest,
                                              max_evals=10,
                                              trials=Trials())
        print(best_run)
        print(best_model)

Answer 1

So it's because you have your parameters not named, let's have a look at this line:

 model.add(LSTM({{choice([32,64,128,256,512])}},W_regularizer=l2({{uniform(0, 1)}})))

as this choice is unnamed - hyperas is scanning function definition and is looking for the parameter name. As it's not named - it assigns the value of the previously named parameter which is output_1 . In order to skip that try:

model.add(LSTM(units={{choice([32,64,128,256,512])}},...)

And do a similar thing with dropout rate:

model.add(Dropout(rate=..))