Tensorflow中多输入模型的优化

Question

I am working with medical data and trying to build a TF model with 2 inputs.
-The first input is a numpy array containing data about several Patient. Some tests are made on each Patient regularly (eg check the weigth of the Patient). They will appear several times in my array.
-The second one is an image of the Patient lungs.

Suppose I have 2 patients named Bob and Jean. Bob has been tested 3 times and Jean has been tested 2 times. I will have 3 rows in the array for Bob and 2 rows for Jean. But for each patient, I have only one picture. So in my DataGenerator, I will have to use this picture 3 times, for each row.
My first input would look like this:

['Name', 'Age', 'Weight']  
['Bob', 54, 80]  
['Bob', 55, 81]  
['Bob', 52, 79]  
['Jean', 40, 90]   
['Jean', 41, 88]  

And the second one like this:

Bob_img = 'Array representing the image of Bob lungs' 
Jean_img = 'Array representing the image of Jean lungs' 

[Picture_array]
[Bob_img]
[Bob_img]
[Bob_img]
[Jean_img]
[Jean_img]

I want to build a CNN to treat the image and then concatenate the output of the CNN with the numpy array. There is a repetition of images in my second input. It uses a lot of RAM and execute the same calculation on the CNN several times. So I would like to know if there is any way to optimize it?

The model is more complex than in the following code but you should have the idea with it.

image_input = K.Input(shape=(512, 512, 60,), name="img_input")
x = tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3), padding='same', activation='relu')(image_input)
x = tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3), padding='same', activation='relu')(x)
x = tf.keras.layers.MaxPool2D(pool_size=(2,2),strides=(2,2))(x)
cnn_output = tf.keras.layers.Flatten()(x)
    
numpy_input = K.Input(shape=(10,), name="numpy_input")
x = tf.keras.layers.concatenate([cnn_output, numpy_input], axis=1)
x = tf.keras.layers.Dense(32, activation='relu')(x)
x = tf.keras.layers.Dense(1, activation='relu')(x)

    
output= tf.keras.layers.Dense(1)(x)
    
model = K.Model(inputs=[image_input, numpy_input], outputs=output)

Plotting the model:

tf.keras.utils.plot_model(model, show_shapes=True)

Link to Model

Answer 1

Finally found the answer: it is possible to implement the training from scratch. The solution is then to forward pass the CNN, then to contenate the CNN forward pass with tabular input and apply the forward pass of the second part of the NN. Finally, I just need to apply the back propagation. I put the link of the guide to write the training from scratch here . And my code if it can be useful:

epochs = 1
for epoch in range(epochs):
    print("\nStart of epoch %d" % (epoch,))
    j=0

    # Iterate over the batches of the dataset.
    for [X1, X2], Y in generator:

        # Open a GradientTape to record the operations run
        # during the forward pass, which enables auto-differentiation.
        with tf.GradientTape() as tape:
            # Forward pass for CNN
            out_imgs = CNN(X2)
            # Replicate the forward pass in the same order than the numpy array
            patients = X1.loc[:, 'Patient'].unique()
            X_imgs = np.empty([X1.shape[0], out_imgs.shape[1]])
            for i in range(len(patients)):
                X_imgs[np.where(X1['Patient']==patients[i])] = out_imgs.numpy()[i]
            # Concatenate the forward pass with the numpy array
            inp_mlp = [tf.keras.layers.concatenate([0.1*X_imgs, np.asarray(X1[temp_SELECTED_COLUMNS])], axis=1)]
            # Forward pass of the second part of the layer
            for layer in model.layers[-3:]:
                inp_mlp.append(layer(inp_mlp[-1]))
            # Calculate loss
            loss = model.compiled_loss(tf.constant(Y), inp_mlp[-1], regularization_losses=model.losses)