Questions tagged [attention]
For questions about the concept of attention in artificial intelligence and machine learning. Attention-like mechanisms were successfully used in natural language processing and computer vision tasks, such as machine translation. For a review of attention-based mechanism used in NLP, take a look at "Attention in Natural Language Processing" by Andrea Galassi et al.
154
questions
0
votes
0
answers
32
views
Why is masked self attention necessary on GPT decoder
I'm currently reading the paper for the first GPT model and I'm confused about why masked self attention is necessary and I haven’t found any good answers online.
The consensus seems to be that we don'...
0
votes
1
answer
70
views
Why is it called multi-headed attention?
Why do we call the attention layer in transformers multi-headed attention when in practice all the attention matrices from different heads (W,K,V) for a single layer are concatenated to perform the ...
- 103
0
votes
0
answers
9
views
Are the outputs of layers in Attention Is All You Need interpretable by mapping to tokens?
In the basic transformer model from 2017, I'm a bit confused what the outputs of each layer are supposed to be. Are they embeddings? If so, does that mean you could examine a given output from a given ...
- 111
0
votes
0
answers
66
views
Why is a linear and not a non-linear transformation used in self-attention to calculate queries, keys and values?
In self-attention, one vector undergoes three different transformations to create the query, keys and values. These are always a simple linear transformations. Why is it considered sufficient? Shouldn'...
- 193
0
votes
2
answers
69
views
Why encoders are required in Transformers
In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....).
I mean even a decoder also has the same ...
- 1
0
votes
0
answers
23
views
What algorithms could I use if I want to increase the accuracy of matched keypoints in an image pair?
Let's say that I used a keypoint detector like SIFT or SuperPoint to detect keypoints in image 1 and 2. Afterwards, I used a keypoint matcher to match corresponding keypoints in this image pair. The ...
- 101
2
votes
1
answer
112
views
Aren't context lengths for transformers an artificial restriction?
Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference.
: https://i....
1
vote
1
answer
77
views
Why is there a shared matrix W in graph attention networks instead of the query-key-value trio like in regular transformers?
In section 2.1 of the Graph attention network paper
The graph attention layer is described as
as an initial step, a shared
linear transformation, parametrized by a weight matrix, W ∈ RF ′×F , is ...
- 35
1
vote
2
answers
111
views
Why is dot-product and not Euclidean distance used for attention?
In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case?
Does Layer ...
- 121
1
vote
2
answers
191
views
Why use a "square root" in the scaled dot product
In attention settings, typically when the both Query Q and Key K are of the same dimension d we can compute their attention score in the following manner:
$$\frac{Q^T K}{\sqrt{d}}$$
The justification ...
- 113
1
vote
0
answers
69
views
Concatenation of Feature vectors in transformers before passing to fcnn
** As I am new to the field , the question might feel little abstract and naïve considering my experience.
I am studying the Transformer architecture and trying to understand the various components ...
0
votes
0
answers
16
views
Multi-instance learning for time-spatio-dependent data
I am trying to use MIL approach from the paper Attention-based Deep Multiple Instance Learning on the data that is a frames of human pose images acquired on different angle on each timepoint (temporal ...
0
votes
0
answers
8
views
Clarification regarding the training status of 'domain specific encoder' in stable diffusion
I am currently studying the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. Specifically, I am focused on the section 3.3. Conditioning Mechanisms.
In ...
- 3,612
1
vote
1
answer
32
views
diagonal of the hypercube in cross-attention
In Cross-attention, divide the similarity between $Q$ and $K$ by $\sqrt{d_k}$.
Here $\sqrt{d_k}$ is the diagonal of the hypercube, is this a coincidence or famous theorem?
$$ \frac{QK^T}{\sqrt{d_k}} $...
1
vote
1
answer
141
views
Why in Multi-Head Attention implementation should we use $3$ linear layers for Q, K, V instead of $3 * h$ layers?
I have been trying to implement a Transformer architecture using PyTorch by following the Attention Is All You Need paper as well as the The Annotated Transformer blog post to compare my code with ...
- 573
7
votes
2
answers
879
views
Can someone help me understand the intuition behind the query, key and value matrices in the transformer architecture?
I have been working mechanically with transformers, hoping that with time clarity about what the query, key, and value matrices represent will develop; but I am still lost. Would greatly benefit from ...
- 501
1
vote
2
answers
127
views
How is the padding mask incorporated in the attention formula?
I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention?
The attention formula taking into ...
- 573
1
vote
0
answers
34
views
Why does averaging attention-weighted positions reduce the effective resolution in transformers?
I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
- 573
0
votes
2
answers
137
views
When do we apply a mask onto our padded values during attention mechanisms
When we are applying a mask onto the padded values in an input sequence, it is typically done through setting the padded values as negative infinity. For example, a tensor of values ...
0
votes
0
answers
26
views
Why use masked attention in the second to 6th decoder blocks during training
For the first block we have access to all the output tokens, so we will have to use masked self-attention. After that in the first decoder block, we have encoder-decoder attention, which will bring ...
2
votes
1
answer
599
views
What if we drop the causal mask in auto-regressive Transformer?
I understand the triangular causal mask in the attention is used to prevent tokens from "looking into the future", but why do we want to prevent that?
Let's suppose we have a model with ...
- 271
0
votes
2
answers
194
views
Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?
My question is why the attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention head is ...
0
votes
1
answer
20
views
Handcraft RNN with attention to extract central element
I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely ...
- 101
5
votes
3
answers
1k
views
Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?
On a high-level temperature and randomness affect the output of a generative language model:
Lower temperature: Produces more focused, conservative, and consistent responses.
Moderate temperature: ...
0
votes
0
answers
27
views
Positional encoding in transformers
After training a single-head transformer, I'm trying to understand some of its internals.
The question I want to answer is:
Given token=X in the last position, how does the query it generates ...
- 101
0
votes
2
answers
386
views
How is a parameter explosion prevented, when connecting a mutlihead attention layer with the dense layers in LLMs (speciafially, LLama)?
I have had a look at LLamas model card, specifically the 7B parameter version:
https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md
which I assume is an encoder only transformer similar ...
- 105
0
votes
0
answers
194
views
Issues with larger context lengths in a transformer model like GPT
Based on my understanding, one of the issues with longer context lengths is the computational complexity of attention mechanism which is quadratic. But is this really a problem on modern hardware with ...
- 101
0
votes
1
answer
90
views
How are the intuitions and mathematics of attention mechanisms related to those of PageRank?
Excuse me if you find this question too vague and not fitting to this forum and feel free to close it. The overall goal of my question is to get a better intuition of the attention concept and ...
0
votes
1
answer
163
views
How Does The Scaled Dot Product's Dimensions Work Out In Mult-Head Attention?
I don't understand how self-attention works with batched values for the $Q \times K^T $ step. According to the diagram below (assume 1 head), once we get past the first 3 linear steps, we arrive at ...
- 21
5
votes
1
answer
1k
views
Why are biases (typically) not used in attention mechanism?
Watching this video implementing attention in a transformer. He set query, key, and value biases to False and said "Typically, people don't use biases for ...
- 554
0
votes
1
answer
308
views
In the attention mechanism, why don't we normalize after multiplying values?
As this question says:
In scaled dot product attention, we scale our outputs by dividing the
dot product by the square root of the dimensionality of the matrix:
The reason why is stated that this ...
- 554
17
votes
1
answer
15k
views
How does the (decoder-only) transformer architecture work?
How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
- 2,083
1
vote
0
answers
88
views
In terms of explainability, is attentive RNN easier to explain than the transformer?
Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
- 191
3
votes
1
answer
358
views
Difference between dot product attention and "matrix attention"
As far as I know, attention was first introduced in Learning To Align And Translate.
There, the core mechanism which is able to disregard the sequence length, is a dynamically-built matrix, of shape ...
- 749
2
votes
0
answers
123
views
How is cross-attention different when you interchange the queries and keys/values?
In the Facebook Paper about Segment Anything, (https://scontent-atl3-2.xx.fbcdn.net/v/t39.2365-6/10000000_6331779526880473_6748528980292947838_n.pdf?_nc_cat=102&ccb=1-7&_nc_sid=3c67a6&...
- 123
3
votes
1
answer
205
views
Machine Translation Transformers: Why Mask in Decoder?
I am trying to understand the purpose of masking in the decoder in the "Attention is All you Need" paper. Why wouldn't we want to rely on bidirectional context when translating? What goes ...
-1
votes
1
answer
151
views
Understanding self attention - How come there is no connection between different states?
During trying to understand transformers by reading Attention is all you need, I noticed the authors constantly refer to "self attention" without explaining it.
The original attention ...
- 749
2
votes
0
answers
59
views
Is it possible for original Vision Transformer (ViT) to do fine-grained semanantic segmentation? if so, how?
As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer.
Without using ...
- 21
0
votes
0
answers
46
views
Why ChatGPT output one token at a time? [duplicate]
My understanding of the language model is that the output of the model is a tensor. So the whole output should be computed all together. But why ChatGPT like models can output one token at a time like ...
- 109
1
vote
1
answer
190
views
How does transformer models like GPT generate valid meaningful response for meaningless garbage input?
My understanding of a transformer model is that it uses the given input to calculate internal query of relate-ness of word meanings, and generate a meaningful response based on its meaning. But if ...
- 67
0
votes
1
answer
74
views
How can I not only classify an intent, but also identify slots and values in it?
I've been working on text -> intent -> command execution for a particular application and while I've found many papers and code that work well for intent classification (1, 2, etc.), they stop ...
- 101
0
votes
1
answer
912
views
Where to find the source code for the research paper "Attention is all you need"?
I am reading "Attention is all you need". I have seen the following link:
Source: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
But when I go to ...
- 99
1
vote
0
answers
69
views
Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?
So I found two contradictory explanations of the MHA (multi-head-self-attention-module):
In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
- 11
0
votes
0
answers
39
views
Why can't traditional neural networks learn to perform the same tasks that attention layers do?
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
1
vote
1
answer
77
views
Are there versions of attention that do not require a key-value pair, but just act on one input?
Are there versions of attention that do not require a key-value pair, but just act on one input? Or does this idea simply not make sense?
0
votes
0
answers
18
views
Does attention in transformers encode any information from positional embeddings?
I know we account for positional embeddings before feeding into attention layers, but would we be able to say that the Q and K dot products intrinsically encode relative positions
- 1
6
votes
2
answers
2k
views
"Attention is all you need" paper : How are the Q, K, V values calculated?
The seminal Attention is all you need paper introduces Transformers and implements the attention mecanism with "queries, keys, values", in an analogy to a retrieval system.
I understand the ...
- 221
0
votes
0
answers
13
views
Can attention models learn statistical parameters such as median, max, mode, mean?
We have some mixed models where after predicting the next word of a sequence, we also want to predict some weights associated to it, related to previous weights of previous words. As the prediction of ...
- 51
0
votes
1
answer
952
views
How to assess if OpenAI's ChatGPT chatbot has a human in the loop? [closed]
I've asked a question and given a couple answers that propose the OpenAI ChatGPT chatbot has humans in the loop (HITL), and that explains the chatbot's extraordinary abilities.
I've been repeatedly ...
- 387
0
votes
0
answers
25
views
Tokenization for treelike structures
I'm pretraining a BERT (bigbird) model to use with SMILES encoding of chemicals. This kind of data is a treelike structure in the form of a string with a single bracket type. Usually this tree isn't ...
