Creating a Keras CNN for image alteration

Question

I'm working on a problem that involves computationally evaluating three-dimensional data of the shape (32, 16, 5) and providing a corrected form of this data also in the shape of (32, 16, 5) . The problem is relatively specific to my field, but it can be viewed as analogous to processing color images (just with five color channels instead of three). If it helps, this could be thought of as a color correction model.

In my initial efforts, I created a random forest model using XGBoost for each of these output parameters. I had good results, but found that the sheer number of output parameters (32*16*5 = 2560) made the runtime of this approach too long, so I am looking for an alternative.

I'm looking at using Keras to solve this, using a convolutional neural.network approach, since the adjacent 'pixels' in my data should have some useful information about their neighbors. Note that 'adjacency' here is both spatial and in the color channels. So far, I am doing alright in creating a simple model that I believe has inputs/outputs of the correct shape, but I am running into an issue when I try to train the model on some dummy images:

#!/usr/bin/env python3
import tensorflow as tf
import pandas as pd
import numpy as np

def create_model(image_shape, batch_size = 10):
    width, height, channels = image_shape
    conv_shape = (batch_size, width, height, channels)
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Conv3D(filters = channels, kernel_size = 3, input_shape = conv_shape, padding = "same"))
    model.add(tf.keras.layers.Dense(channels, activation = "relu"))
    return model


if __name__ == "__main__":
    image_shape = (32, 16, 5)

    # Create test input/output data sets:
    input_img = np.random.rand(*image_shape) # Create one dummy input image
    output_img = np.random.rand(*image_shape) # Create one dummy output image

    # Create a bogus 'training set' by copying the input/output images into lists many times
    inputs = [input_img]*500
    outputs = [output_img]*500

    # Create the model and fit it to the dummy data
    model = create_model(image_shape)
    model.summary()
    model.compile(loss = "mean_squared_error", optimizer = "adam", metrics = ["accuracy"])
    model.fit(input_img, output_img)

However, when I run this code, I get the following error:

ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=5, found ndim=3. Full shape received: [32, 16, 5]

I am not really sure what the other two expected dimensions are for the data passed into model.fit() . I suspect this is a problem with the way that I am formatting my input data. Even if I have a list of input/output images, that will only bring the ndim of my data to 4, not 5.

I have been trying to find similar examples in the documentation and around the web to see what I'm doing incorrectly, but 3D convolution on a non-classifier.network seems a bit off the beaten path, and I'm not having much luck (or just don't know the name of what I should search for).

I have tried passing the dummy training set to model.fit instead of two individual images. Fitting with model.fit(inputs, outputs) instead, I get:

ValueError: Layer sequential expects 1 inputs, but it received 500 input tensors.

It seems that passing a list of tensors isn't correct here. If I convert the list of input images to numpy arrays with:

inputs = np.array(inputs)
outputs = np.array(outputs)

This does bring up the number of dimensions in my input data to 4, but Keras is still expecting 5. The error I get in this case is very similar to the first:

ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=5, found ndim=4. Full shape received: [None, 32, 16, 5]

I'm definitely not understanding something here, and any help would be appreciated.

Answer 1

I think you made two mistakes in your code:

1. Instead of using Conv3D , you need to use Conv2D .
2. model.fit(input_img, output_img) should be model.fit(inputs, outputs) .

The reason why you need to use Conv2D is the shape of your data is (length,width,channel) , it doesn't possess an extra dimension.

Try the script below

#!/usr/bin/env python3
import tensorflow as tf
import pandas as pd
import numpy as np

def create_model(image_shape, batch_size = 10):
    width, height, channels = image_shape
    conv_shape = (width, height, channels)
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Conv2D(filters = channels, kernel_size = 3, input_shape = conv_shape, padding = "same"))
    model.add(tf.keras.layers.Dense(channels, activation = "relu"))
    return model


if __name__ == "__main__":
    image_shape = (32, 16, 5)

    # Create test input/output data sets:
    input_img = np.random.rand(*image_shape) # Create one dummy input image
    output_img = np.random.rand(*image_shape) # Create one dummy output image

    # Create a bogus 'training set' by copying the input/output images into lists many times
    inputs = np.array([input_img]*500)
    outputs = np.array([output_img]*500)

    # Create the model and fit it to the dummy data
    model = create_model(image_shape)
    model.summary()
    model.compile(loss = "mean_squared_error", optimizer = "adam", metrics = ["accuracy"])
    model.fit(inputs, outputs)