theano error from keras

Question

I am running a keras script (no direct call to theano in my script) and I get the following error:

TypeError: ('An update must have the same type as the original shared                     
variable (shared_var=<TensorType(float32, matrix)>, 
shared_var.type=TensorType(float32, matrix),     
update_val=Elemwise{add,no_inplace}.0, 
update_val.type=TensorType(float64, matrix)).', 
'If the difference is related to the broadcast pattern, 
you can call the tensor.unbroadcast(var, axis_to_unbroadcast[, ...])
function to remove broadcastable dimensions.')

I have seen the error from folks running theano directly, but not through keras. Not sure what I should do, since I am not dealing with tensors directly.

Answer 1

the problem was that there is a change in keras version (I am currently using keras 0.3.2 with theano 0.8.0) and what used to be fine does not work well with he new keras version.

The following was the original code, and see the fix below.

from keras.models import Sequential
import keras.optimizers
from keras.layers.core import Dense, Dropout
from keras.layers.normalization import BatchNormalization
from keras.layers.advanced_activations import PReLU
from keras.layers.core import Activation
from keras.optimizers import SGD, Adam

from sklearn.preprocessing import StandardScaler
from sklearn.base import BaseEstimator, RegressorMixin

class NnRegression(BaseEstimator, RegressorMixin):
        def __init__(self, apply_standart_scaling=True,
             dropx=[0.2, 0.5, 0.5], nb_neuronx=[50, 30], nb_epoch=105, validation_split=0.,
             verbose=1):
        self.apply_standart_scaling = apply_standart_scaling
        self.dropx = dropx
        self.nb_neuronx = nb_neuronx
        self.nb_epoch = nb_epoch
        self.validation_split = validation_split
        self.verbose = verbose

    def fit(self, X, y):

        nb_features = X.shape[1]
        self.standart_scaling = StandardScaler() if self.apply_standart_scaling else None

        if self.standart_scaling:
            X = self.standart_scaling.fit_transform(X)

        model = Sequential()
        model.add(Dropout(input_shape = (nb_features,),p= self.dropx[0]))
        model.add(Dense(output_dim = self.nb_neuronx[0], init='glorot_uniform'))
        model.add(PReLU())
        model.add(BatchNormalization(self.nb_neuronx[0],)))
        model.add(Dropout(self.dropx[1]))

        model.add(Dense(self.nb_neuronx[1], init='glorot_uniform'))
        model.add(PReLU())
        model.add(BatchNormalization(self.nb_neuronx[0],)))
        model.add(Dropout(self.dropx[2]))

        model.add(Dense(1, init='glorot_uniform'))

        nn_verbose = 1 if self.verbose>0 else 0
        optz = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08)
        model.compile(optimizer=Adam(),loss='mse')
        model.fit(X, y, batch_size=16,
              nb_epoch=self.nb_epoch, validation_split=self.validation_split, verbose=nn_verbose)

        self.model = model

    def predict(self, X):
        if self.standart_scaling:
            X = self.standart_scaling.transform(X)
        return self.model.predict_proba(X, verbose=0)

well, it turns out that the problem is this single line of code:

model.add(BatchNormalization(self.nb_neuronx[0],)))

It should actually be:

model.add(BatchNormalization())

because the number of neurons has no business within the normalization layer (however this did not bother in a previous keras version).

This apparently causes theano to generate new weights that are not float32 but float64, and that triggers the message above.