1
$\begingroup$

In the original paper of transformers they using positional encoding to capture the position of each word in the sentence and for calculate that it using sin and cos ,like shom in the image.Sinusoidal positional encoding In Bert and the author architecture that based on transformers ,they use learnable position encoding ,so they initialize the vectors of positional encoding randomly and then start to adjust them in training.
My question is why we use the learnable positional encoding, what is the objective?

$\endgroup$

1 Answer 1

1
$\begingroup$

transformers models use the sinusoidal position encoding to capture the position of each token, and this method has some limits :

  1. the sinusoidal position encoding is manually designed so it is not flexible enough and does not contain any learnable parameters
  2. if we have a long sequence so we may have the same vector position because sin and cos are periodic function, so we need unique position vector for each token and the perfect way to do that is using learnable position encoding
  3. and this position encoding is independent on dataset, sometimes if we change the position of one token in sentence there is no different in meaning but sometimes the meaning of sentence change, so it depend on dataset, for that the position encoding should be learnable from data
    so this is the main principle issue that we have with the traditional method of position encoding and for that many of researcher try to introduce some solution and use the learnable position encoding, this paper may help better understand the logic behind this idea http://proceedings.mlr.press/v119/liu20n/liu20n.pdf
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .