fondamental question about regularization techniques to solve overfitting problem in neural networks

Question

I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset.

Using some classical regularization techniques (dropout=0.5) and weight_decay=0.7, I have an accuracy of 84% on the training dataset and 70% on the validation dataset.

My question is: Do regularization techniques usually improve the validation accuracy? Or they will only reduce the training accuracy to a closer level to the validation accuracy?

As my objective is improve the validation accuracy to 90%, and I am wondering if there is any hope that solving the overfitting problem would increase validation accuracy, so should I continue investigating on the regularization techniques or thinking about changing the whole model.

Answer 1

Regularization techniques reduce overfitting. This is why they tend to reduce training accuracy when applied to a model: they prevent the model to learn noise from the training data. For the same reason the validation accuracy increases: learning less noise means better generalization on unseen data. These two aspects are sides of the same coin.

There would be no point in developing a technique that only reduce training accuracy to match for aesthetic reasons the validation one. So yes, you can keep investigating regularization techniques and you should always use them. But be aware that since you have a strict goal you want to achieve you might also want to try other architectures or expand your dataset. Regularization will mostly help you to get a better grasp on the best scores you can achieve (if you get 84% accuracy on training, I argue you'll probably never get to 90% validation, unless the model was far from converging) and they boost a bit good results, but they will not help you making a huge jump up.