Following-up my question about my over-fitting network
I have tried several things :
- Simplify the architecture
- Apply more (and more !) Dropout
- Data augmentation
But I always reach similar results : training accuracy is eventually going up, while validation accuracy never exceed ~70%.
I think I simplified enough the architecture / applied enough dropout, because my network is even too dumb to learn anything and return random results (3-classes classifier => 33% is random accuracy), even on training dataset :
My question is : This accuracy of 70% is the best my model can reach ?
If yes :
- Why the training accuracy reach such high scores, and why so fast, knowing this architecture seems to be not compatible ?
- My only option to improve the accuracy is then to change my model, right ?
If no :
- What are my options to improve this accuracy ?
I'v tried a bunch of hyperparameters, and a lot of time, depending of these parameters, the accuracy does not change a lot, always reaching ~70%. However I can't exceed this limit, even though it seems easy to my network to reach it (short convergence time)
Here is the Confusion matrix :
I don't think the data or the balance of the class is the problem here, because I used a well-known / explored dataset : SNLI Dataset
And here is the learning curve :
Note : I used accuracy instead of error rate as pointed by the resource of Martin Thoma
It's really ugly one. I guess there is some problem here. Maybe the problem is that I used the result after 25 epoch for every values. So with little data, training accuracy don't really have time to converge to 100% accuracy. And for bigger training data, as pointed in earlier graphs, the model overfit so the accuracy is not the best one.