6 votes
Accepted

What is the difference between the positional encoding techniques of the Transformer and GPT?

The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token ...
spiridon_the_sun_rotator's user avatar
4 votes

Which positional encoding BERT use?

BERT uses trained position embeddings. The original paper does not say it explicitly, the term position embeddings (as opposed to encoding) suggests it is trained. When you look at BERT layers in ...
Jindřich's user avatar
  • 391
3 votes

What is the intuition behind position-encoding?

I understand the confusion. Although transformers are autoregressive (they predict something based on past information), they are not recurrent (do not have hidden states). In fact, think of a ...
Robin van Hoorn's user avatar
3 votes
Accepted

Is Positional Encoding always needed for using Transformer models correctly?

Positional Encodings in Transformers exist to give the model some information about the position of the embedding. This makes sense in fields like NLP or Time Series Data, since the position(order) ...
DKDK's user avatar
  • 329
2 votes

Positional Encoding of Time-Series features

I think your input should have shape [batch_size, time_points, n_features] which would correspond to ...
Tom Huntington's user avatar
2 votes

Are positional embeddings computed during or before training?

Yes, your reading of the paper is correct. Vectors generated from sinusoid functions are fixed, and are not modified during training. There exists an alternative variant -- initialize vectors as ...
user2907934's user avatar
1 vote
Accepted

Why is the sinusoidal model classified as absolute positional encoding in some literature?

Absolute position embeddings capture the absolute location of a token. Absolute location would refer to e.g., the 1st, 2nd, 3rd token etc. The sinusoidal embeddings in Vaswani's paper capture this ...
Alexander Wan's user avatar
1 vote

Why use exponential and log in Positional Encoding of Transformer

The current code just implements the now-standard expressions for positional embeddings given in the original transformer paper (Attention is all you need). In section 3.5 of this paper they suggest ...
just another mathmo's user avatar
1 vote

Is there a notion of location in Transformer architecture in subsequent self-attention layers?

Note that the architecture of the encoder (and decoder) uses residual connections that add the original input to the output of the self attention layer. Thus, the information from the positional ...
pi-tau's user avatar
  • 807

Only top scored, non community-wiki answers of a minimum length are eligible