I'm fine tuning a Bert/Roberta model for a classification task. As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout layers. Is this a good idea?
The usual practice is to the first token embedding as an input to the classifier, which forces the last layer to collect the relevant information from the previous layers to this particular embeddings.
You might probably view it as wasting the potential of the last layer, where most hidden states are just ignored. From this point of view, an additional attention layer might better use what BERT already knows. On the other hand, adding an entirely new layer means new parameters that need to be learned from scratch, so more data is needed. Because this is not usually done, I guess the gain from it is probably very small.
I would rather focus on more traditional ways of improving classification accuracy: cleaning/preprocessing data and better regularization.