如何使用两个 DataIterator(s) 训练 CNN model 的双流输入？

Question

I'm building a convolutional neural network (CNN) model consisting of dual stream image data input of 'RGB' channels and 'grayscale' channel converging into singular stream of shape (None, width, height, 4*C) , then Dense() .

With big dataset, I must / forced to utilise <class 'ImageDataGenerator'> :

from tensorflow.keras.preprocessing.image import ImageDataGenerator 

datagen = ImageDataGenerator()

and flow the dataset via directory:

train_C = datagen.flow_from_directory(
    ... , 
    color_mode = 'rgb', 
    ...
)

train_gr = datagen.flow_from_directory(
    ... , 
    color_mode = 'grayscale', 
    ... 
)

Unfortunately, to train the model with fit method as the following :

model.fit( x = [input_1, input_2], ... )

it raised the following error:

ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of types {"<class 'keras_preprocessing.image.directory_iterator.DirectoryIterator'>"}), <class 'NoneType'>

So, how should I resolve this issue?

Note: Python 3.X & Tensorflow 2.X

Answer 1

1. Prepare your combined train generator: train_generator = zip(train_C, train_gr)
2. Concatenate the output of last layers: fused_1 = concatenate([averagepool, maxpool])
3. Create conv or other dense layers: top_model = Dense(4096, activation='relu',name="dense_one")(fused_1)
4. Create final model: model = Model(inputs=[model1.input, model2.input], outputs=[top_model])
5. Fit the model: history = model.fit_generator(train_generator, ...)