How to improve testing accuracy when training accuracy is high?

Question

Following-up my question about my over-fitting network

My deep neural network is over-fitting :

I have tried several things :

• Simplify the architecture
• Apply more (and more !) Dropout
• Data augmentation

But I always reach similar results : training accuracy is eventually going up, while validation accuracy never exceed ~70%.

I think I simplified enough the architecture / applied enough dropout, because my network is even too dumb to learn anything and return random results (3-classes classifier => 33% is random accuracy), even on training dataset :

My question is : This accuracy of 70% is the best my model can reach ?

If yes :

• Why the training accuracy reach such high scores, and why so fast, knowing this architecture seems to be not compatible ?
• My only option to improve the accuracy is then to change my model, right ?

If no :

• What are my options to improve this accuracy ?

I'v tried a bunch of hyperparameters, and a lot of time, depending of these parameters, the accuracy does not change a lot, always reaching ~70%. However I can't exceed this limit, even though it seems easy to my network to reach it (short convergence time)

Edit

Here is the Confusion matrix :

I don't think the data or the balance of the class is the problem here, because I used a well-known / explored dataset : SNLI Dataset

And here is the learning curve :

Note : I used accuracy instead of error rate as pointed by the resource of Martin Thoma

It's really ugly one. I guess there is some problem here. Maybe the problem is that I used the result after 25 epoch for every values. So with little data, training accuracy don't really have time to converge to 100% accuracy. And for bigger training data, as pointed in earlier graphs, the model overfit so the accuracy is not the best one.

Answer 1

I identified the origin of this overfitting..

Origins

I tried a lot of models, putting more and more dropout, simplifying as much as I could.

No matter what I did, after a few epoch of good learning, invariably my loss function was going up. I tried simpler and simpler models, always the same overfitting behavior. What bugged me at that moment is that no matter what kind of model I used, how deep or how complex, always the accuracy was fine, stabilized at some nice level.

So I tried the simplest model I could imagine : Input => Dense with 3 hidden units => Output. Finally I got random results, with a 33% accuracy ! From here, I guilt again my network, layer by layer, to see which one was causing the overfitting.

And it was the Embedding layer.

Even with a simple network like Input => Embeddings => Dense with 3 hidden units => Output, the model was overfitting.

How to solve it

In Keras, simply instantiate the Embeddings layer with trainable=False. After doing this, no more overfit.

In my opinion, this is quite counter-intuitive : I want my embeddings to evolve with the data I show to the network. But look like I can't...

Answer 2

I think sometimes it can also help to examine your test and training sets. Fundamentally, your data was produced by an underlying process/system that has certain properties. The system can have many "states" and all the possible states form the state space. If you have really tried things like dropout and regularization, my guess would be that the test set is somehow different from your train set. It is possible that your training set only takes samples from one part of the state space (AKA, your samples might all be similar in the training set and the test set has different samples - imagine you are classifying humans and all of your training samples have a class label of 1 meaning all the training samples have humans in them -> and all your test samples have no humans in them Good luck with that!). Some questions to ask:

1. Are you combining datasets from different sources? If so: If you have "n" sources of data, you need to make sure that your training set has many samples from each of the "n" sources of data and your test set has samples from each of the "n" sources.

2. Are you shuffling your data enough and randomly putting samples in both the training and test sets? This relates to the human example I gave, make sure your training set has a little bit of everything (different combinations of inputs and/or outputs) and your testing set has a little bit of everything (different combinations of inputs and/or outputs).