Questions tagged [positional-encoding]
The positional-encoding tag has no usage guidance.
14
questions
0
votes
0
answers
18
views
How to interpret CNN output image? CNN expressive enough for particle positions?
Let us suppose we have a squared grid and some particles (active vertices) that are distributed on it.
We can construct a 2D image, where basically the input image is a matrix with 0s (no particle) ...
- 31
0
votes
0
answers
12
views
Would convolution filters provide better position encoding than the traditional cos/sin functions with k,n,i, and d?
Would convolution filters provide better position encoding than the traditional cos/sin functions with k,n,i and d? My thoughts are that the traditional position encoding functions introduce too much ...
0
votes
1
answer
810
views
Which positional encoding BERT use?
It is a little bit confusing that someone is explaining that BERT is using sinusoidal functions for BERT position encoding and someone is saying BERT just uses absolute position.
I checked that ...
- 101
2
votes
1
answer
1k
views
Positional Encoding of Time-Series features
I’m trying to use a Transformer Encoder I coded with weather feature vectors which are basically 11 features about the weather in the dimension ...
- 21
0
votes
1
answer
35
views
Are positional embeddings computed during or before training?
I'm trying to practically frame the concept of positional embeddings as introduced in the original paper.
As far as I've understood, what we do is basically creating some other vectors in addition to ...
- 277
0
votes
1
answer
315
views
Is Positional Encoding always needed for using Transformer models correctly?
I am trying to make a model that uses a Transformer to see the relationship between several data vectors, but the order of the data is not relevant in this case, so I am not using the Positional ...
- 191
3
votes
1
answer
47
views
Is there a notion of location in Transformer architecture in subsequent self-attention layers?
Transformer architecture (without position embedding) is by the very construction equivariant to the permutation of tokens. Given query $Q \in \mathbb{R}^{n \times d}$ and keys $K \in \mathbb{R}^{n \...
- 2,174
2
votes
0
answers
195
views
Is there any point in adding the position embedding to the class token in Transformers?
The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches.
Is there any point in doing so?
The purpose of introduction ...
- 2,174
2
votes
0
answers
355
views
Positional Encoding in Transformer on multi-variate time series data hurts performance
I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data.
As I just experiment with the positional encoding portion of the code I ...
- 121
0
votes
0
answers
172
views
Has positional encoding been used in convolutional layers?
Positional encoding (PE) is an essential part of the self-attention layers in the transformer architectures since without adding it in some way (fixed of learnable) to the input embeddings model has ...
- 2,174
7
votes
2
answers
1k
views
What is the difference between the positional encoding techniques of the Transformer and GPT?
I know the original Transformer and the GPT (1-3) use two slightly different positional encoding techniques.
More specifically, in GPT they say positional encoding is learned. What does that mean? ...
- 285
2
votes
0
answers
103
views
How does positional encoding work in the transformer model?
In the transformer model, to incorporate positional information of texts, the researchers have added a positional encoding to the model. How does positional encoding work? How does the positional ...
- 1,006
1
vote
0
answers
76
views
Why do both sine and cosine have been used in positional encoding in the transformer model?
The Transformer model proposed in "Attention Is All You Need" uses sinusoid functions to do the positional encoding.
Why have both sine and cosine been used? And why do we need to separate the odd ...
- 11
2
votes
0
answers
228
views
How do the sine and cosine functions encode position in the transformer?
After going through both the "Illustrated Transformer" and "Annotated Transformer" blog posts, I still don't understand how the sinusoidal encodings are representing the position of elements in the ...
- 121