0
$\begingroup$

I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data.

As I just experiment with the positional encoding portion of the code I set up a toy model: I generated a time series that contains the log changes of a sine function and run a classification model that predicts whether the subsequent value is positive or negative. Simple enough. I also added a few time series with random walks to try to throw off the model.

Predictably, the model very quickly reaches a categorical accuracy of around 99%. Without positional encoding that happens already in the 3rd epoch. However, with positional encoding (I use the same implementation as proposed in the "Attention is all you need" paper), it takes over 100 epochs to reach a similar accuracy level.

So, clearly, all else being equal, learning with positional encoding takes much longer to reach an equal accuracy level than without positional encoding.

Has anyone witnessed similar observations? Apparently adding the positional encodings to the actual values seems to confuse the model. I have not tried concatenations yet. Any advice?

Edit: Or does it simply mean that learned positional encodings perform better than sin/cos encodings? I have not made any special provisions to encourage learned positional encodings, I simply either added the positional encodings to the actual values or I did not.

$\endgroup$
2
  • $\begingroup$ I wonder what kind of positional encoding you used. If you just add, like I think the Transformer paper did, then it makes perfect sense that it's harder to train because that basically scrambles all the word vectors based on position. Perhaps concatenating could work better, but you'd have more weights to learn. $\endgroup$
    – user253751
    Aug 11 at 8:34
  • $\begingroup$ @user253751, which then begs the question of the need for postional encodings. I definitely reach higher accuracy levels and that a lot earlier without positional encodings as applied to multivariate time series $\endgroup$
    – Matt
    Aug 11 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.