在pyspark中使用带有辍学的Keras序列化模型

Question

I have several neural networks built using Keras that I used so far mostly in Jupyter. I often save models from scikit-learn with joblib and Keras with json + hdf5 and use them in other notebooks without issue.

I made a Python Spark application that can make use of those serialized models in cluster mode. joblib models are working fine however, I encountered an issue with Keras.

Here is the model used in notebook and pyspark:

def build_gru_model():
    model = Sequential()
    model.add(Embedding(max_nb_words, 128, input_length=max_sequence_length, dropout=0.2))
    model.add(GRU(128, dropout_W=0.2, dropout_U=0.2))
    model.add(Dense(2, activation='softmax'))
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

both called the same way:

preds = model.predict_proba(data, verbose=0)

However, only in Spark I get the error:

MissingInputError: ("An input of the graph, used to compute DimShuffle{x,x,x,x}(keras_learning_phase), was not provided and not given a value.Use the Theano flag exception_verbosity='high',for more information on this error.", keras_learning_phase)

I've done the mandatory search and found: https://github.com/fchollet/keras/issues/2430 which points to https://keras.io/getting-started/faq/

If I indeed remove dropout from my model, it works. However, I fail to understand how to implement something that would allow me to keep dropout during the training phase like described in the FAQ.

Based on the model code, how one would accomplish this?

Answer 1

You can try to put (before your prediction)

import keras.backend as K
K.set_learning_phase(0)

It should set your learning phase to 0 (test time)