How to combine specific CNN models that work better at slightly different tasks?

Question

I'm not sure how to describe this in the most accurate way but I'll give it a shot.

I've developed a Inception-Resnet V2 model for detecting audio signals via spectrogram. It does a pretty good job but is not exactly the way I'd like it to be.

Some details: I use 5 sets of data to evaluate my model during training. They are all similar but slightly different. Once I get to a certain threshold of F1 Scores for each training set I stop training. My overall threshold is pretty hard to get to. Every time training develops a model that produces a "best yet" of one of these data sets I save the model.

What I've noticed is that, during training, some round will produce a high F1 Score for one particular set while the other sets languish as mediocre. Then, several dozen rounds later, another data set will peak while the others are mediocre. Overall the entire model gets better but there are always some models that work better for some data sets.

What I would like to know is, given I might have 5 different models that each work better for a particular subset of data, is there a way that I can combine these models (either as a whole or better yet their particular layers) to produce a single model that works the best for all my data validation subsets?

Thank you. Mecho

Answer 1

What you're describing are "Ensemble Models" -- where multiple models are trained in parallel, and then combined at inference time to squeeze out better performance.

This article gives a decent overview: https://towardsdatascience.com/ensemble-models-5a62d4f4cb0c

A single algorithm may not make the perfect prediction for a given dataset. Machine learning algorithms have their limitations and producing a model with high accuracy is challenging. If we build and combine multiple models, the overall accuracy could get boosted.

And they're also covered in Stanford's 2017 deep learning computer vision course, lecture 7: https://youtu.be/_JB0AO7QxSA?t=3098.

Rather than having just one model, we’ll train 10 different models independently from different initial random restart. At test time, we’ll run our data through all of the 10 models and average the predictions.

I recommend you check out this course, since it also delves quite a bit into convolutional networks too.