Questions tagged [attention]

For questions about the concept of attention in artificial intelligence and machine learning. Attention-like mechanisms were successfully used in natural language processing and computer vision tasks, such as machine translation. For a review of attention-based mechanism used in NLP, take a look at "Attention in Natural Language Processing" by Andrea Galassi et al.

Filter by
Sorted by
Tagged with
0 votes
0 answers
32 views

Why is masked self attention necessary on GPT decoder

I'm currently reading the paper for the first GPT model and I'm confused about why masked self attention is necessary and I haven’t found any good answers online. The consensus seems to be that we don'...
Kiran Manicka's user avatar
0 votes
1 answer
70 views

Why is it called multi-headed attention?

Why do we call the attention layer in transformers multi-headed attention when in practice all the attention matrices from different heads (W,K,V) for a single layer are concatenated to perform the ...
Tarique's user avatar
  • 103
0 votes
0 answers
9 views

Are the outputs of layers in Attention Is All You Need interpretable by mapping to tokens?

In the basic transformer model from 2017, I'm a bit confused what the outputs of each layer are supposed to be. Are they embeddings? If so, does that mean you could examine a given output from a given ...
Stan Shunpike's user avatar
0 votes
0 answers
66 views

Why is a linear and not a non-linear transformation used in self-attention to calculate queries, keys and values?

In self-attention, one vector undergoes three different transformations to create the query, keys and values. These are always a simple linear transformations. Why is it considered sufficient? Shouldn'...
Kasia's user avatar
  • 193
0 votes
2 answers
69 views

Why encoders are required in Transformers

In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....). I mean even a decoder also has the same ...
Swastik's user avatar
0 votes
0 answers
23 views

What algorithms could I use if I want to increase the accuracy of matched keypoints in an image pair?

Let's say that I used a keypoint detector like SIFT or SuperPoint to detect keypoints in image 1 and 2. Afterwards, I used a keypoint matcher to match corresponding keypoints in this image pair. The ...
user402016's user avatar
2 votes
1 answer
112 views

Aren't context lengths for transformers an artificial restriction?

Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference. : https://i....
Robert Wegner's user avatar
1 vote
1 answer
77 views

Why is there a shared matrix W in graph attention networks instead of the query-key-value trio like in regular transformers?

In section 2.1 of the Graph attention network paper The graph attention layer is described as as an initial step, a shared linear transformation, parametrized by a weight matrix, W ∈ RF ′×F , is ...
oliver.c's user avatar
1 vote
2 answers
111 views

Why is dot-product and not Euclidean distance used for attention?

In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case? Does Layer ...
Betterthan Kwora's user avatar
1 vote
2 answers
191 views

Why use a "square root" in the scaled dot product

In attention settings, typically when the both Query Q and Key K are of the same dimension d we can compute their attention score in the following manner: $$\frac{Q^T K}{\sqrt{d}}$$ The justification ...
Sheed's user avatar
  • 113
1 vote
0 answers
69 views

Concatenation of Feature vectors in transformers before passing to fcnn

** As I am new to the field , the question might feel little abstract and naïve considering my experience. I am studying the Transformer architecture and trying to understand the various components ...
Buddha Dev Bhattacharjee's user avatar
0 votes
0 answers
16 views

Multi-instance learning for time-spatio-dependent data

I am trying to use MIL approach from the paper Attention-based Deep Multiple Instance Learning on the data that is a frames of human pose images acquired on different angle on each timepoint (temporal ...
PatrickHellman's user avatar
0 votes
0 answers
8 views

Clarification regarding the training status of 'domain specific encoder' in stable diffusion

I am currently studying the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. Specifically, I am focused on the section 3.3. Conditioning Mechanisms. In ...
hanugm's user avatar
  • 3,612
1 vote
1 answer
32 views

diagonal of the hypercube in cross-attention

In Cross-attention, divide the similarity between $Q$ and $K$ by $\sqrt{d_k}$. Here $\sqrt{d_k}$ is the diagonal of the hypercube, is this a coincidence or famous theorem? $$ \frac{QK^T}{\sqrt{d_k}} $...
diffusion stable's user avatar
1 vote
1 answer
141 views

Why in Multi-Head Attention implementation should we use $3$ linear layers for Q, K, V instead of $3 * h$ layers?

I have been trying to implement a Transformer architecture using PyTorch by following the Attention Is All You Need paper as well as the The Annotated Transformer blog post to compare my code with ...
Daviiid's user avatar
  • 573
7 votes
2 answers
879 views

Can someone help me understand the intuition behind the query, key and value matrices in the transformer architecture?

I have been working mechanically with transformers, hoping that with time clarity about what the query, key, and value matrices represent will develop; but I am still lost. Would greatly benefit from ...
Chinmay's user avatar
  • 501
1 vote
2 answers
127 views

How is the padding mask incorporated in the attention formula?

I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention? The attention formula taking into ...
Daviiid's user avatar
  • 573
1 vote
0 answers
34 views

Why does averaging attention-weighted positions reduce the effective resolution in transformers?

I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
Daviiid's user avatar
  • 573
0 votes
2 answers
137 views

When do we apply a mask onto our padded values during attention mechanisms

When we are applying a mask onto the padded values in an input sequence, it is typically done through setting the padded values as negative infinity. For example, a tensor of values ...
synphonics's user avatar
0 votes
0 answers
26 views

Why use masked attention in the second to 6th decoder blocks during training

For the first block we have access to all the output tokens, so we will have to use masked self-attention. After that in the first decoder block, we have encoder-decoder attention, which will bring ...
Souvik Mandal's user avatar
2 votes
1 answer
599 views

What if we drop the causal mask in auto-regressive Transformer?

I understand the triangular causal mask in the attention is used to prevent tokens from "looking into the future", but why do we want to prevent that? Let's suppose we have a model with ...
nalzok's user avatar
  • 271
0 votes
2 answers
194 views

Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?

My question is why the attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention head is ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
20 views

Handcraft RNN with attention to extract central element

I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely ...
Daraan's user avatar
  • 101
5 votes
3 answers
1k views

Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?

On a high-level temperature and randomness affect the output of a generative language model: Lower temperature: Produces more focused, conservative, and consistent responses. Moderate temperature: ...
Hans-Peter Stricker's user avatar
0 votes
0 answers
27 views

Positional encoding in transformers

After training a single-head transformer, I'm trying to understand some of its internals. The question I want to answer is: Given token=X in the last position, how does the query it generates ...
Lem0n's user avatar
  • 101
0 votes
2 answers
386 views

How is a parameter explosion prevented, when connecting a mutlihead attention layer with the dense layers in LLMs (speciafially, LLama)?

I have had a look at LLamas model card, specifically the 7B parameter version: https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md which I assume is an encoder only transformer similar ...
user2741831's user avatar
0 votes
0 answers
194 views

Issues with larger context lengths in a transformer model like GPT

Based on my understanding, one of the issues with longer context lengths is the computational complexity of attention mechanism which is quadratic. But is this really a problem on modern hardware with ...
rahul's user avatar
  • 101
0 votes
1 answer
90 views

How are the intuitions and mathematics of attention mechanisms related to those of PageRank?

Excuse me if you find this question too vague and not fitting to this forum and feel free to close it. The overall goal of my question is to get a better intuition of the attention concept and ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
163 views

How Does The Scaled Dot Product's Dimensions Work Out In Mult-Head Attention?

I don't understand how self-attention works with batched values for the $Q \times K^T $ step. According to the diagram below (assume 1 head), once we get past the first 3 linear steps, we arrive at ...
rkuang25's user avatar
5 votes
1 answer
1k views

Why are biases (typically) not used in attention mechanism?

Watching this video implementing attention in a transformer. He set query, key, and value biases to False and said "Typically, people don't use biases for ...
Peyman's user avatar
  • 554
0 votes
1 answer
308 views

In the attention mechanism, why don't we normalize after multiplying values?

As this question says: In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this ...
Peyman's user avatar
  • 554
17 votes
1 answer
15k views

How does the (decoder-only) transformer architecture work?

How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
Robin van Hoorn's user avatar
1 vote
0 answers
88 views

In terms of explainability, is attentive RNN easier to explain than the transformer?

Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
hH1sG0n3's user avatar
  • 191
3 votes
1 answer
358 views

Difference between dot product attention and "matrix attention"

As far as I know, attention was first introduced in Learning To Align And Translate. There, the core mechanism which is able to disregard the sequence length, is a dynamically-built matrix, of shape ...
Gulzar's user avatar
  • 749
2 votes
0 answers
123 views

How is cross-attention different when you interchange the queries and keys/values?

In the Facebook Paper about Segment Anything, (https://scontent-atl3-2.xx.fbcdn.net/v/t39.2365-6/10000000_6331779526880473_6748528980292947838_n.pdf?_nc_cat=102&ccb=1-7&_nc_sid=3c67a6&...
a6623's user avatar
  • 123
3 votes
1 answer
205 views

Machine Translation Transformers: Why Mask in Decoder?

I am trying to understand the purpose of masking in the decoder in the "Attention is All you Need" paper. Why wouldn't we want to rely on bidirectional context when translating? What goes ...
gumbelgrumbelgumbel's user avatar
-1 votes
1 answer
151 views

Understanding self attention - How come there is no connection between different states?

During trying to understand transformers by reading Attention is all you need, I noticed the authors constantly refer to "self attention" without explaining it. The original attention ...
Gulzar's user avatar
  • 749
2 votes
0 answers
59 views

Is it possible for original Vision Transformer (ViT) to do fine-grained semanantic segmentation? if so, how?

As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer. Without using ...
wanburana's user avatar
0 votes
0 answers
46 views

Why ChatGPT output one token at a time? [duplicate]

My understanding of the language model is that the output of the model is a tensor. So the whole output should be computed all together. But why ChatGPT like models can output one token at a time like ...
Bin Wang's user avatar
  • 109
1 vote
1 answer
190 views

How does transformer models like GPT generate valid meaningful response for meaningless garbage input?

My understanding of a transformer model is that it uses the given input to calculate internal query of relate-ness of word meanings, and generate a meaningful response based on its meaning. But if ...
BlueSnake's user avatar
0 votes
1 answer
74 views

How can I not only classify an intent, but also identify slots and values in it?

I've been working on text -> intent -> command execution for a particular application and while I've found many papers and code that work well for intent classification (1, 2, etc.), they stop ...
Ani's user avatar
  • 101
0 votes
1 answer
912 views

Where to find the source code for the research paper "Attention is all you need"?

I am reading "Attention is all you need". I have seen the following link: Source: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf But when I go to ...
Vy Do's user avatar
  • 99
1 vote
0 answers
69 views

Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?

So I found two contradictory explanations of the MHA (multi-head-self-attention-module): In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
AAT's user avatar
  • 11
0 votes
0 answers
39 views

Why can't traditional neural networks learn to perform the same tasks that attention layers do?

If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things: Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
tunafriedrice's user avatar
1 vote
1 answer
77 views

Are there versions of attention that do not require a key-value pair, but just act on one input?

Are there versions of attention that do not require a key-value pair, but just act on one input? Or does this idea simply not make sense?
postnubilaphoebus's user avatar
0 votes
0 answers
18 views

Does attention in transformers encode any information from positional embeddings?

I know we account for positional embeddings before feeding into attention layers, but would we be able to say that the Q and K dot products intrinsically encode relative positions
Manny's user avatar
  • 1
6 votes
2 answers
2k views

"Attention is all you need" paper : How are the Q, K, V values calculated?

The seminal Attention is all you need paper introduces Transformers and implements the attention mecanism with "queries, keys, values", in an analogy to a retrieval system. I understand the ...
Soltius's user avatar
  • 221
0 votes
0 answers
13 views

Can attention models learn statistical parameters such as median, max, mode, mean?

We have some mixed models where after predicting the next word of a sequence, we also want to predict some weights associated to it, related to previous weights of previous words. As the prediction of ...
arivero's user avatar
  • 51
0 votes
1 answer
952 views

How to assess if OpenAI's ChatGPT chatbot has a human in the loop? [closed]

I've asked a question and given a couple answers that propose the OpenAI ChatGPT chatbot has humans in the loop (HITL), and that explains the chatbot's extraordinary abilities. I've been repeatedly ...
yters's user avatar
  • 387
0 votes
0 answers
25 views

Tokenization for treelike structures

I'm pretraining a BERT (bigbird) model to use with SMILES encoding of chemicals. This kind of data is a treelike structure in the form of a string with a single bracket type. Usually this tree isn't ...
Materia Gravis's user avatar