Questions tagged [positional-encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-1
votes
0answers
25 views

Why some encoders reduce dimensionality of data while some others increase it? [closed]

We know that the encoder part in VAE reduces dimensions of data. In some other papers related to neural radiance fields and 3d shape reconstruction like https://arxiv.org/pdf/2106.01553.pdf, the ...
0
votes
1answer
45 views

Is Positional Encoding always needed for using Transformer models correctly?

I am trying to make a model that uses a Transformer to see the relationship between several data vectors, but the order of the data is not relevant in this case, so I am not using the Positional ...
1
vote
1answer
27 views

Is there a notion of location in Transformer architecture in subsequent self-attention layers?

Transformer architecture (without position embedding) is by the very construction equivariant to the permutation of tokens. Given query $Q \in \mathbb{R}^{n \times d}$ and keys $K \in \mathbb{R}^{n \...
1
vote
0answers
29 views

Is there any point in adding the position embedding to the class token in Transformers?

The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches. Is there any point in doing so? The purpose of introduction ...
0
votes
0answers
16 views

What are the pros and cons of using a normal positional encoding in an adjacency matrix?

I understand that a normal positional encoding helps a transformer to understand pictures better and that it allows the (otherwise permutational invariant transformer-network) to create relationships ...
2
votes
0answers
57 views

Positional Encoding in Transformer on multi-variate time series data hurts performance

I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data. As I just experiment with the positional encoding portion of the code I ...
0
votes
0answers
40 views

Has positional encoding been used in convolutional layers?

Positional encoding (PE) is an essential part of the self-attention layers in the transformer architectures since without adding it in some way (fixed of learnable) to the input embeddings model has ...
2
votes
1answer
46 views

What is the difference between the positional encoding techniques of the Transformer and GPT?

I know the original Transformer and the GPT (1-3) use two slightly different positional encoding techniques. More specifically, in GPT they say positional encoding is learned. What does that mean? ...
2
votes
0answers
58 views

How does positional encoding work in the transformer model?

In the transformer model, to incorporate positional information of texts, the researchers have added a positional encoding to the model. How does positional encoding work? How does the positional ...
1
vote
0answers
57 views

Why do both sine and cosine have been used in positional encoding in the transformer model?

The Transformer model proposed in "Attention Is All You Need" uses sinusoid functions to do the positional encoding. Why have both sine and cosine been used? And why do we need to separate the odd ...
2
votes
0answers
177 views

How do the sine and cosine functions encode position in the transformer?

After going through both the "Illustrated Transformer" and "Annotated Transformer" blog posts, I still don't understand how the sinusoidal encodings are representing the position of elements in the ...