Is there a theory behind which model is good for a classification task for the convolutional neural network?

Question

Let say I'm trying to apply CNN for image classification. There are lots of different models to choose and we can try an ensemble, but given a limit amount of resources, it does not allow to try everything.

Is there a theory behind which model is good for a classification task for the convolutional neural network?

Right now, I'm just taking an average of three predictions.

predictions_model = [y_pred_xceptionAug,y_pred_Dense121_Aug,y_pred_resnet50Aug]
predictions = np.mean(predictions_model,axis=0)

But each model's performance is different. Is there better way for ensemble methods?

Answer 1

Ensembles aren't very popular in the field of computer vision. The main reason why this is, is that models are already so large parameter-wise that it is hard to fit multiple models in-memory for classification. Since there are effective ways of training very large models, people would rather create a larger networks if they had the capacity than averaging the results from multiple ones.

That being said, there is no reason why ensembling wouldn't have beneficial results for your task.

One way would be, as you do, to average the results of the models. This is usually used to reduce the bias of weaker models. Another way would be to use meta-modelling, i.e. create a fourth model (even as simple as a linear classifier) that will be trained with the outputs of the three CNNs as its input features. The idea is that the meta-model will learn the best way to weight the outputs of the CNNs so that, instead of them all having an equal vote (as is the case when you average them), the meta-model will learn the best way to weigh them.