Cannot reproduce results with Keras

Question

I'm facing a mistery with keras. I'm following the example shown in the book "Practical machine learning with python".

I execute the following commands

    import keras
    from sklearn.datasets import load_breast_cancer
    cancer = load_breast_cancer()
    X_train = cancer.data[:340]
    y_train = cancer.target[:340]
    X_test = cancer.data[340:]
    y_test = cancer.target[340:]
    from keras.models import Sequential
    from keras.layers import Dense, Dropout
    model = Sequential()
    model.add(Dense(15, input_dim=30, activation='relu'))
    model.add(Dense(15, activation='relu'))
    model.add(Dense(15, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
    model.fit(X_train, y_train, epochs=20, batch_size=50)

This is the output (I am pasting only accuracy and loss) of the last line

    loss: 0.4130 - acc: 0.8500

    loss: 0.2211 - acc: 0.9147

    loss: 0.4366 - acc: 0.8441

    loss: 0.2292 - acc: 0.9118

    loss: 0.2547 - acc: 0.9029

    loss: 0.2399 - acc: 0.8941

    loss: 0.4907 - acc: 0.8618

    loss: 0.3180 - acc: 0.8853

    loss: 0.2452 - acc: 0.9029

    loss: 0.4381 - acc: 0.8588

    loss: 0.2242 - acc: 0.9029

    loss: 0.2700 - acc: 0.8941

    loss: 0.2022 - acc: 0.9118

    loss: 0.3545 - acc: 0.8559

    loss: 0.3242 - acc: 0.8882

    loss: 0.2504 - acc: 0.9147

    loss: 0.5074 - acc: 0.8676

    loss: 0.1980 - acc: 0.9353

    loss: 0.3794 - acc: 0.8882

    loss: 0.2567 - acc: 0.9000

Then I repeat the same steps for the same model with a different name

    model2 = Sequential()
    model2.add(Dense(15, input_dim=30, activation='relu'))
    model2.add(Dense(15, activation='relu'))
    model2.add(Dense(15, activation='relu'))
    model2.add(Dense(1, activation='sigmoid'))
    model2.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
    model2.fit(X_train, y_train, epochs=20, batch_size=50)

This is the output of the last line (again, I will show only acc. and loss)

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

     loss: 7.3616 - acc: 0.5382

Why I don't get the same result (a much greater accuracy and a much lesser loss)?

Thank you very much

Answer 1

That is totally normal.

Model weights are initialized randomly. Sometime a model starts with better luck than another.

Since you're using "relu" and just a few neurons (15), there is also the possibility of all your relu outputs being zero, and this the backpropagation cannot happen. This results in a frozen model, like in your second example.

For using "relu" activations, a proper weight initialization and a proper learning rate must be selected to avoid this effect. (I'm not an expert in this part, though).

If you want to see reproductible results, you need to set a constant random seed before you run the code. (And probably you will need to restart your python kernel whenever you want to create a new model)

import random
random.seed(chooseAnInteger)

Answer 2

I solved by specifying a kernel_initializer

model.add(Dense(15, activation='relu', kernel_initializer='random_uniform'))

Thank everyone for the help