如何将 1 个图像分配给每个颜色通道作为 ResNet50 的输入

Question

I am trying to build a ResNet50 model that performs regression to a single output and takes an input of 3 images. All 3 images are the same size (brain scans of 160*160) and I want to assign each image to a color channel (first image to Red, second to Green, and third to Blue, as all 3 images are black and white.), in order to avoid building a new network. I have already build a regressor with ResNet50 which accepts 1 image as input. Here is my code:

def load_train(path):
    
    """
    It loads the train part of dataset from path
    """
    #1 / 255
    labels = pd.read_csv('C:\\Users\\sumad\\OneDrive - San José Unified School District\\Documents\\AMYLOID DATASET COMPLETE ALL\\Full Set\\AllAmyloidREFNORMTarget.csv')
    train_datagen = ImageDataGenerator(validation_split=0.2, rescale=None)
    train_gen_flow = train_datagen.flow_from_dataframe(
        dataframe=labels,
        directory='C:\\Users\\sumad\OneDrive - San José Unified School District\\Documents\\AMYLOID DATASET COMPLETE ALL\\Full Set\\AllAmyloidOneImagePNG\\',
        x_col='ID',
        y_col='Value',
        target_size=(224, 224),
        batch_size=32,
        class_mode='raw',
        subset = 'training',
        seed=12345)

    return train_gen_flow


def load_test(path):
    
    """
    It loads the validation/test part of dataset from path
    """
    labels = pd.read_csv('C:\\Users\\sumad\\OneDrive - San José Unified School District\\Documents\\AMYLOID DATASET COMPLETE ALL\\Full Set\\AllAmyloidREFNORMTarget.csv')
    validation_datagen = ImageDataGenerator(validation_split=0.2, rescale=None)
    test_gen_flow = validation_datagen.flow_from_dataframe(
    dataframe = labels,
    directory='C:\\Users\\sumad\OneDrive - San José Unified School District\\Documents\\AMYLOID DATASET COMPLETE ALL\\Full Set\\AllAmyloidOneImagePNG\\',
    x_col="ID",
    y_col="Value", 
    class_mode="raw", 
    target_size=(224,224), 
    batch_size=32,
    subset = "validation",
    seed=1234,
    )

    return test_gen_flow


def create_model(input_shape):
    
    # we will use ResNet50 architecture, with freezing top layers
    backbone = ResNet50(input_shape=input_shape, weights='imagenet', include_top=False)
    model = Sequential()
    model.add(backbone)
    
    model.add(Dropout(0.3))
    model.add(GlobalAveragePooling2D())
    
    model.add(Dense(1, activation='linear'))
    optimizer = Adam(learning_rate=0.0003)
    model.compile(optimizer=optimizer, loss='mae', metrics=['mae'])
    print(model.summary())

    return model

def train_model(model, train_data, test_data, batch_size=32, epochs=100,
                steps_per_epoch=None, validation_steps=None):

    history = model.fit(train_data, validation_data=test_data, batch_size=batch_size, 
              epochs=epochs, steps_per_epoch=steps_per_epoch, 
              validation_steps=validation_steps, verbose=2)

    # Get training and test loss histories
    training_loss = history.history['loss']
    test_loss = history.history['val_loss']

    epoch_count = range(1, len(training_loss) + 1)

    plt.plot(epoch_count, training_loss, 'r--')
    plt.plot(epoch_count, test_loss, 'b-')
    plt.legend(['Training Loss', 'Test Loss'])
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.show();

    return model

def main():

    path = 'C:\\Users\\sumad\\OneDrive - San José Unified School        District\\Documents\\AMYLOID DATASET COMPLETE ALL\\Full Set\\'

    train_data = load_train(path)
    test_data = load_test(path)

    model = create_model(input_shape = (224, 224, 3))

model = train_model(model, train_data, test_data)

Answer 1

Since you are already using keras, I assume you have also Pillow installed.

This code snippet stacks three images, read as grayscale, into a single one as RGB. Then what you need is to call this method for each example to generate a list of RGB-like images and its labels, but for this poin I'ld need more information on your specific settings, and as I see it, is out of scope for this question.

import numpy as np
from PIL import Image

def load_images_to_single(path_a, path_b, path_c, size=(224, 224):

    # load images as grayscale
    img_a = np.array(Image.open(path_a).convert('L').resize(size))
    img_b = np.array(Image.open(path_b).convert('L').resize(size))
    img_c = np.array(Image.open(path_c).convert('L').resize(size))

    # map images to channels as:
    #     img_a -> R channel
    #     img_b -> G channel
    #     img_c -> B channel
    img_rgb = np.dstack([img_a, img_b, img_c])

    return img_rgb

The all you need to do is, first, iterate over the dataset to build your X_train, y_train and the same for test, since this beheaviour is a little different from default.

It will only work is your data fits into memory, otherwise you need to implement your own DataGeneration .