Can we create an ensemble of deep learning models without increasing the classification time?

Question

I want to improve my ResNet model by creating an ensemble of X number of this model, taking the X best one I have trained. For what I've seen, a technique like bagging will take X time longer to classify an image, which is really not an option in my case.

Is there a way to create an ensemble without increasing the required classifying time? Note that I don't care about increasing the training time, because it only needs to be done one time, compared to the classification which could be made a very large number of time.

Answer 1

There is no magic pill for doing what you want. Extra computation cannot come free.

So one way this can be achieved is by using multiple worker machines to run inference in parallel.
Each model could run on a different machine using tensorflow serving .

For every new inference do the following:

• Have a primary machine which takes up the job of running the inference
• This primary machine, submits requests to different workers (all of which can run in parallel)
• The primary machine collects results from each individual worker, and creates the final output by combining them based upon your ensemble logic.

Answer 2

Depends on the ensembling method; it's an active area of research I suggest you look into, but I'll provide some examples below:

• Dropout : trains parts of the model at any given iterations, thus effectively training a multi-NN ensemble
• Weights averaging : train X models on X different splits of data to learn different features, average the early-stopped weights (requires advanted treatment)
• Lookahead optimizer : automates the above by performing the averaging during training
• Parallel weak learners : run X models, but each model taking 1/X the time to process - which can be achieved by eg inserting a strides=X convolutional layer at input; best starting bet is at X=2, so you'll average two models' predictions at output, each prediction made in parallel (which can run faster than original single model)

If you have a multi-core CPU, however, multi-model ensembling shouldn't pose much of a problem, as per last bullet, you can run inference concurrently, so inference time shouldn't increase much

---

More on parallelism : if a single model is large enough, CPU parallelism will no longer help - you'll also need to ensure multiple models can fit in memory (RAM). The alternative then is again a form of downsampling to cut computation