使用不同输入的集合模型

Question

I instantiated 3 simple not so deep models(could be deeper as I like) and trained them with 3 seperate inputs. To be clear, the first input is the face image, second is eyes and the third is mouth (for facial expression recognition). I'd like to ensemble a model that I feed with 3 inputs (with all same class label ofc) and get a single label output. The motivation is that 90% accurate models may perform better when ensembled together. It should look like this:

Facial input-----------Eyes Input------------Mouth Input
 (100x100x1)          (100x100x1)          (100x100x1) 
     |                     |                     | 

   .....                 ......                .....

     |                     |                     |
      ___________________________________________
                 Some concatenation over here
                           |
                   | Output Label|    

Or should I complete forget this; acquire the response of bottom Dense layers of each model after test, and combine them to a feature vector and classify. Clueless over here...

Answer 1

Suppose your models are called model1 , model2 and model3 . You can use the functional API to merge your models like this:

input1 = model1.input
input2 = model2.input
input3 = model3.input

m1 = model1(input1)
m2 = model2(input2)
m3 = model3(input3)

output = concatenate([m1, m2, m3]) # this is the concatenation
output = Dense(10)(output) # just an example for what you could add to the model
output = Activation('softmax')(output)

model = Model(inputs=[input1,input2,input3], outputs=output)

model can then be trained and it will train all three models simultaniously. If you only want to train the part of the model on top of the three models, you can use the trainable parameter.