Questions tagged [attention]
For questions about the concept of attention in artificial intelligence and machine learning. Attention-like mechanisms were successfully used in natural language processing and computer vision tasks, such as machine translation. For a review of attention-based mechanism used in NLP, take a look at "Attention in Natural Language Processing" by Andrea Galassi et al.
122
questions
0
votes
0
answers
12
views
How does transformer models like GPT generate valid meaningful response for meaningless garbage input?
My understanding of a transformer model is that it uses the given input to calculate internal query of relate-ness of word meanings, and generate a meaningful response based on its meaning. But if ...
- 45
0
votes
0
answers
5
views
Going beyond intent classification
I've been working on text -> intent -> command execution for a particular application and while I've found many papers and code that work well for intent classification (1, 2, etc.), they stop ...
- 101
0
votes
1
answer
35
views
Where to find the source code for the research paper "Attention is all you need"?
I am reading "Attention is all you need". I have seen the following link:
Source: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
But when I go to ...
1
vote
0
answers
10
views
Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?
So I found two contradictory explanations of the MHA (multi-head-self-attention-module):
In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
- 11
0
votes
0
answers
25
views
Why can't traditional neural networks learn to perform the same tasks that attention layers do?
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
1
vote
1
answer
39
views
Are there versions of attention that do not require a key-value pair, but just act on one input?
Are there versions of attention that do not require a key-value pair, but just act on one input? Or does this idea simply not make sense?
0
votes
0
answers
11
views
Does attention in transformers encode any information from positional embeddings?
I know we account for positional embeddings before feeding into attention layers, but would we be able to say that the Q and K dot products intrinsically encode relative positions
- 1
5
votes
1
answer
119
views
"Attention is all you need" paper : How are the Q, K, V values calculated?
The seminal Attention is all you need paper introduces Transformers and implements the attention mecanism with "queries, keys, values", in an analogy to a retrieval system.
I understand the ...
- 191
0
votes
0
answers
11
views
Can attention models learn statistical parameters such as median, max, mode, mean?
We have some mixed models where after predicting the next word of a sequence, we also want to predict some weights associated to it, related to previous weights of previous words. As the prediction of ...
- 51
0
votes
1
answer
469
views
How to assess if OpenAI's ChatGPT chatbot has a human in the loop? [closed]
I've asked a question and given a couple answers that propose the OpenAI ChatGPT chatbot has humans in the loop (HITL), and that explains the chatbot's extraordinary abilities.
I've been repeatedly ...
- 331
0
votes
0
answers
18
views
Tokenization for treelike structures
I'm pretraining a BERT (bigbird) model to use with SMILES encoding of chemicals. This kind of data is a treelike structure in the form of a string with a single bracket type. Usually this tree isn't ...
3
votes
1
answer
390
views
What's the relationship between number of heads and embedding dimension in Transformers?
I am reading the book: Natural Language Processing with Transformers. It has the following paragraph
Although head_dim does not have to be smaller than the number of embedding dimensions of the ...
- 460
0
votes
2
answers
38
views
Why do the values in the cross attentional mechanism within a transformer come from the encoder and not from the decoder?
The transformer architecture contains a cross attention mechanism which is enriching the encoder with information from the decoder. The place where this takes place is visualized in the image below:
...
- 373
0
votes
0
answers
13
views
Adding MNIST images by using them as channel inputs
I'm trying to create a generative neural network that can offer "basic sum" mathematical solutions using the MNIST dataset from a conditional input.
I've curated a dataset of MNIST examples ...
- 1
3
votes
1
answer
206
views
What is the intuition behind self-attention?
I've been watching a few lectures on transformers, especially for language translation, though it seemingly becomes more confusing the more I watch.
In this lecture, there seems to be two conflicting ...
- 133
1
vote
0
answers
12
views
Where exactly is permutation happening in equation 5 of the paper "Learning with Sets in Multiple Instance Regression Applied to Remote Sensing"?
I am reading the article Learning with Sets in Multiple Instance Regression Applied to Remote Sensing about creating an embedding which is order-invariant to inputs ($m_{l}$). They referred to order-...
- 23
1
vote
1
answer
38
views
Group image classification for whether containing unrelated images
I'm kind of new to computer vision, and wondering whether this is any existing researches / solutions to following scenarios.
Suppose I have a dataset, each data point contains a few images (< 20 ...
- 11
0
votes
0
answers
59
views
1
vote
2
answers
69
views
How does Seq2Seq with attention actually use the attention (i.e. the context vector)?
For neural machine translation, there's this model "Seq2Seq with attention", also known as the "Bahdanau architecture" (a good image can be found on this page), where instead of ...
- 111
0
votes
0
answers
17
views
How to get sentence from embedding vector with Universal Sentence Encoder?
Given a sentence embedding vector from a Sentence Encoder (like Sentence-BERT), I want to train a model to generate the original sentence (list of word embedding).
Are there any architectures to ...
2
votes
0
answers
30
views
Using similarity score within lstm embedding for attention based mechanism
Yesterday, I found this fascinating paper about predicting various clinical conditions using an attention based LSTM. I don't have any practical experience with attention mechanism or transformers, ...
- 21
0
votes
0
answers
20
views
How to Decide on the Structure of a Neural Network for Time Series Forecasting?
Apologies for the noob question. I am attempting time series forecasting (with a combination of lag and categorical features) using tensorflow and struggling to find an optimal combination of RNN/LSTM ...
- 111
3
votes
1
answer
72
views
What is the most important predecessor of the transformer model?
I'm wondering what the origins of the transformer as proposed in Attention Is All You Need are. The paper itself provides some interesting pointers to the literature on self-attention such as:
A ...
- 373
0
votes
0
answers
34
views
Does this kind of attention exist?
As someone who is new to deep learning, I am only familiar with self-attention.
I'm designing a model. Imagine there are n data, which the $i_{th}$ data can be represented as a vector $x_i$. And the ...
- 121
5
votes
2
answers
624
views
Why are embeddings added, not concatenated?
Let's consider the following example from BERT
I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". ...
- 181
2
votes
0
answers
264
views
Why use a fully connected layer for attention?
In the paper Neural Machine Translation by Jointly Learning to Align and Translate, attention is used with a single fully connected layer. Specifically, in the auto-regressive set up (equation 4), the ...
- 121
0
votes
0
answers
113
views
Is it possible to use attention in non sequential data in neural networks?
I'm still trying to understand the attention mechanism. It is not clear to me what query, key, and value mean yet, for example.
However, my main issue is regarding how to apply attention in my use ...
- 101
5
votes
1
answer
2k
views
When exactly does the split into different heads in Multi-Head-Attention occur?
I am confused by the Multi-Head part of the Multi-Head-Attention used in Transformers. My question concerns the implementations in Pytorch of nn.MultiheadAttention and its forward method ...
- 53
1
vote
0
answers
91
views
What does "position" in "each position in the decoder" denote in the Transformer's original paper?
I am reading Attention is All You Need and I feel confused about the word "position" in this paper, by the way I'm not native English speaker which may cause my confusion which has confused ...
- 11
1
vote
0
answers
64
views
When training a seq2seq model is it better to train using the models outputs or expected outputs?
When training any seq2seq model you have a target and a source. The source may be a sentence ...
- 1,316
0
votes
0
answers
47
views
What are attention heads in a Graph Attention Layers
I am using the EGATConv layer for an edge classification task. One of the constructor's parameters is num_heads, which is number of attention heads.
I can't really ...
0
votes
0
answers
643
views
Is attention always better then an RNN/CNN?
We've all read the attention is all you need paper, but is it really all you need? Can you effectively replace any RNN/CNN with an attention transformer and see better results?
- 1,316
0
votes
1
answer
454
views
Does it make sense to add an additional attention layer while fine-tuning Bert?
I'm fine tuning a Bert/Roberta model for a classification task.
As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout ...
0
votes
1
answer
56
views
Attention: Isn't it redundant to apply a linear layer to both the keys and values?
Transformer attention is calculated $Attention(X) =X W^V\times \text{columnwise-softmax} (Att(X))$ where the attention attention matrix is $$Att(X) = Q \times K = {X W}^Q \times ({X W}^K)^T = {X W}^Q (...
- 278
0
votes
0
answers
26
views
Why is it common to use K=V in Attention layers?
Context:
Hi I recently read from the keras docs: "key: Optional key Tensor of shape (B, S, dim). If not given, will use value for both key and value, which is the most common case."
I found ...
- 306
0
votes
0
answers
129
views
How to use copy mechanism and attention together?
Is the copy mechanism and attention related for a Neural Machine Translation task when source and target vocabulary are the same?
Copy mechanism means unknown words would be copied from source to the ...
- 153
2
votes
2
answers
51
views
Is this aggregation of multiple convolutions of the same input a type of attention or dynamic convolution?
Are there any examples of people performing multiple convolutions at a single depth and then performing feature max aggregation as a convex combination as a form of "dynamic convolutions"?
...
- 165
0
votes
1
answer
54
views
Sensible integer embedding/encoding for distinguishing elements of a set?
I am trying to train a model that takes in a set of feature vectors (which comes with an ID to uniquely identify elements of the set) and outputs a target for each element in the set (in a permutation-...
- 101
2
votes
1
answer
591
views
What is multi-head attention doing mathematically, and how is it different from self-attention?
I'm trying to understand the difference between the concept of self-attention and multi-head attention. The latter is not actually too clear to me.
I understand that, in the case of self-attention, we ...
- 277
2
votes
1
answer
1k
views
What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining.
I'm trying to understand the shapes used during decoder (both self-...
- 171
3
votes
1
answer
47
views
Is there a notion of location in Transformer architecture in subsequent self-attention layers?
Transformer architecture (without position embedding) is by the very construction equivariant to the permutation of tokens. Given query $Q \in \mathbb{R}^{n \times d}$ and keys $K \in \mathbb{R}^{n \...
- 2,154
1
vote
0
answers
41
views
Why does research on faster Transformers focus on the query-key product?
A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism:
$$\text{softmax}\left(\frac{Q K^T}{\sqrt{d}} \right)V,$$
As I understand it, the runtime,...
- 111
3
votes
2
answers
688
views
In layman terms, what does "attention" do in a transformer?
I heard from many people about the paper titled Attention Is All You Need by Ashish Vaswani et al.
What actually does the "attention" do in simple terms? Is it a function, property, or some ...
- 3,481
0
votes
1
answer
68
views
Is the multi-head attention in the transformer a weighted adjacency matrix?
Are multi-head attention matrices weighted adjacency matrices?
The job of the multi-head-attention mechanism in transformer models is to determine how likely a word is to appear after another word. In ...
- 93
1
vote
0
answers
70
views
Are there any benefits of adding attention to linear layers?
Is attention useful only in transformer/convolution layers? Can I add it to linear layers? If yes, how (on a conceptual level, not necessarily the code to implement the layers)?
- 13
5
votes
0
answers
1k
views
Is there a proper initialization technique for the weight matrices in multi-head attention?
Self-attention layers have 4 learnable tensors (in the vanilla formulation):
Query matrix $W_Q$
Key matrix $W_K$
Value matrix $W_V$
Output matrix $W_O$
Nice illustration from https://jalammar....
- 2,154
5
votes
1
answer
3k
views
Why does a transformer not use an activation function following the multi-head attention layer?
I was hoping someone could explain to me why in the transformer model from the "Attention is all you need" paper there is no activation applied after both the multihead attention layer and ...
- 103
1
vote
0
answers
115
views
Couldn't the self-attention mechanism be replaced with a global depth-wise convolution?
The main advantages of the self-attention mechanism are:
Ability to capture long-range dependencies
Ease to parallelize on GPU or TPU
However, I wonder why the same goals cannot be achieved by ...
- 2,154
0
votes
1
answer
71
views
What is the bit memory task?
I learned from this post about the so-called bit memory:
They froze its self-attention and feed-forward layers and, in separate copies, fine-tuned peripheral layers on each on a wide range of tasks: ...
- 857
1
vote
1
answer
163
views
Attention mechanism: Why apply multiple different transformations to obtain query, key, value
I have two questions about the structure of attention modules:
Since I work with imagery I will be talking about using convolutions on feature maps in order to obtain attention maps.
If we have a set ...
- 111