Ensemble with voting in deep learning models

Question

I am working on a multimodal deep learning classifiers with RGB-D images. i have developed two seperate models for each case. The first one is a LSTM with CNN in the begining for the RGB images with shape (3046,200,200,3) , and the second one is an LSTM for the depth images with shape (3046,200,200) . I'm trying to figure out how i can do Ensemble-with-voting.

I have done the bellow code tha finds the most voted labels but seems not working well. Is there any another way to do late fusion to the models?

labels = []
loaded_model = keras.models.load_model('Saved/LSTM_RGB/LSTMRGB.h5')
predictsRGB = np.argmax(loaded_model.predict(x_testRGB), axis=1)
labels.append(predictsRGB)

loaded_model = keras.models.load_model('Saved/LSTM_DEPTH/LSTMDEPTH.h5')
predictsDepth = np.argmax(loaded_model.predict(x_testDepth), axis=1)
labels.append(predictsDepth)

print(labels)
# Ensemble with voting
labels = np.array(labels)
labels = np.transpose(labels, (1, 0))
print(labels,labels.shape)
labels = scipy.stats.mode(labels)[0]
labels = np.squeeze(labels)
print(labels)
print(labels.shape)

Answer 1

You have two classifier, in that cases directly voting does not make sense because how to resolve the ties?

Since you are doing np.argmax on models' prediction, I believe the models output probabilities as a prediction. If you can not introduce a third model, you can average out these probabilities and then take argmax.

As an example: Model 1 says class 1 with prob 0.9 and class 0 with probability 0.1 whereas model 2 says class 1 with probability 0.3 and class 0 with 0.7.

The averaged probabilities would be: Class1: 0.6 Class0: 0.4 and you would go for class1. In a way, it makes sense since model 1 was much more sure about its prediction than the model2