0
$\begingroup$

I'm fine tuning a Bert/Roberta model for a classification task. As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout layers. Is this a good idea?

$\endgroup$
1
  • $\begingroup$ What method are you using for fine tuning? Do you have your code in GitHub? $\endgroup$
    – Cloud Cho
    Commented Nov 20, 2023 at 22:12

2 Answers 2

0
$\begingroup$

The usual practice is to the first token embedding as an input to the classifier, which forces the last layer to collect the relevant information from the previous layers to this particular embeddings.

You might probably view it as wasting the potential of the last layer, where most hidden states are just ignored. From this point of view, an additional attention layer might better use what BERT already knows. On the other hand, adding an entirely new layer means new parameters that need to be learned from scratch, so more data is needed. Because this is not usually done, I guess the gain from it is probably very small.

I would rather focus on more traditional ways of improving classification accuracy: cleaning/preprocessing data and better regularization.

$\endgroup$
2
  • $\begingroup$ Thanks, i will focus on this. $\endgroup$ Commented Apr 29, 2022 at 10:19
  • $\begingroup$ Please, add information on "where most hidden states are just ignored". $\endgroup$
    – Cloud Cho
    Commented Nov 20, 2023 at 22:14
0
$\begingroup$

A few points to consider:

  • Is your dataset big enough for fine-tuning BERT model, a large model might overfit your dataset, and using a small model might help in that case
  • Also instead of a whole attention block, try adding only a linear layer to see if that helps, you can also reduce the number of parameters in the layer by using a bottleneck architecture, i.e. adding 2 layers of shape (N x N/4) & (N/4 x N).
  • Instead of full fine-tuning, try using LoRA where you can control the rank of matrices according to the size of your dataset.
$\endgroup$
1
  • $\begingroup$ Please, share any reference of LoRA and its experiment. $\endgroup$
    – Cloud Cho
    Commented Nov 20, 2023 at 22:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .