Keras CNN for Image transformation, NAN loss

Question

I am writing a CNN for for an image transformation (usually done with fourier transform and set of parameters)

I am having trouble with my loss function returning NAN from the first epoch, I've tried changing my optimizers, adjusting learning rate, adding dropout and batch normalization, changing the design of my model but nothing seems to be working.

I've also ensure my data has no nan of inf in itself. My data set does contain a large amount of zeros so im not sure if this is affecting it. I have normalized the data to be in range [0,1]

even if I simplify it to only have this layer the loss is nan right away (# channels = 1, image size 1024x1024):

model.add(Conv2D(1, 1 , activation = 'relu', input_shape = (1,1024, 1024),
                 kernel_regularizer=regularizers.l2(0.01),
                 bias_regularizer=regularizers.l2(0.01))) 

opt = Adagrad(learning_rate = 0.0001, clipvalue = .5)
model.compile(optimizer = opt, loss = 'mean_squared_error',
          metrics= ['acc', 'mse'])

history = model.fit_generator(generator=training_generator,
                steps_per_epoch = 30, verbose = 2,
                epochs = 20)

I am using my own generator to feed batches of the image data to the model. I'm currently using a batch size of only 2.

The output is currently just:

    Epoch 1/20
    - 62s - loss: nan - acc: 0.2009 - mse: nan
    Epoch 2/20
   - 61s - loss: nan - acc: 0.2007 - mse: nan
   Epoch 3/20
   - 62s - loss: 9.5592 - acc: 0.1675 - mse: 0.2778
   Epoch 4/20
   - 61s - loss: nan - acc: 0.2340 - mse: nan
   Epoch 5/20
   - 61s - loss: nan - acc: 0.1172 - mse: nan
   Epoch 6/20
   - 61s - loss: nan - acc: 0.1173 - mse: nan

and so on

The problem I am trying to model is am image transformation to a hologram using fourier transform and phase shift, the output image should have the same dimensions as the input (1, 1024, 1024)

Any help or insight would be greatly appreciated, maybe I am just missing a very obvious step!

Answer 1

I think your problem comes from the way you defined your model :

• You are using the mean_squarred_error loss, so you are calculating the squarred error of every pixel on your image !
• Moreover, your are using relu activation, so your model can predict values between 0 and inf.
• Those two point combinated : image of 1024 * 1024 with a mean error of 50 makes you a loss so hudge that it goes NaN.

Give us more precisions about what you are trying to predicting if you want more help on what kind of activation/loss using !