I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset.
Using some classical regularization techniques (dropout=0.5) and weight_decay=0.7, I have an accuracy of 84% on the training dataset and 70% on the validation dataset.
My question is: Do regularization techniques usually improve the validation accuracy? Or they will only reduce the training accuracy to a closer level to the validation accuracy?
As my objective is improve the validation accuracy to 90%, and I am wondering if there is any hope that solving the overfitting problem would increase validation accuracy, so should I continue investigating on the regularization techniques or thinking about changing the whole model.