Questions tagged [attention]

For questions about the concept of attention in artificial intelligence and machine learning. Attention-like mechanisms were successfully used in natural language processing and computer vision tasks, such as machine translation. For a review of attention-based mechanism used in NLP, take a look at "Attention in Natural Language Processing" by Andrea Galassi et al.

Filter by
Sorted by
Tagged with
3 votes
1 answer
48 views

Why are embeddings added, not concatenated?

Let's consider the following example from BERT I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". ...
user avatar
  • 161
2 votes
0 answers
40 views

Why use a fully connected layer for attention?

In the paper Neural Machine Translation by Jointly Learning to Align and Translate, attention is used with a single fully connected layer. Specifically, in the auto-regressive set up (equation 4), the ...
user avatar
  • 121
0 votes
0 answers
30 views

Is it possible to use attention in non sequential data in neural networks?

I'm still trying to understand the attention mechanism. It is not clear to me what query, key, and value mean yet, for example. However, my main issue is regarding how to apply attention in my use ...
user avatar
  • 101
2 votes
1 answer
92 views

When exactly does the split into different heads in Multi-Head-Attention occur?

I am confused by the Multi-Head part of the Multi-Head-Attention used in Transformers. My question concerns the implementations in Pytorch of nn.MultiheadAttention and its forward method ...
user avatar
0 votes
0 answers
25 views

What does "position" in "each position in the decoder" denote in the Transformer's original paper?

I am reading Attention is All You Need and I feel confused about the word "position" in this paper, by the way I'm not native English speaker which may cause my confusion which has confused ...
user avatar
  • 1
1 vote
0 answers
17 views

When training a seq2seq model is it better to train using the models outputs or expected outputs?

When training any seq2seq model you have a target and a source. The source may be a sentence ...
user avatar
  • 1,344
0 votes
0 answers
24 views

What are attention heads in a Graph Attention Layers

I am using the EGATConv layer for an edge classification task. One of the constructor's parameters is num_heads, which is number of attention heads. I can't really ...
user avatar
0 votes
0 answers
70 views

Is attention always better then an RNN/CNN?

We've all read the attention is all you need paper, but is it really all you need? Can you effectively replace any RNN/CNN with an attention transformer and see better results?
user avatar
  • 1,344
0 votes
1 answer
52 views

Does it make sense to add an additional attention layer while fine-tuning Bert?

I'm fine tuning a Bert/Roberta model for a classification task. As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout ...
user avatar
0 votes
1 answer
28 views

Attention: Isn't it redundant to apply a linear layer to both the keys and values?

Transformer attention is calculated $Attention(X) =X W^V\times \text{columnwise-softmax} (Att(X))$ where the attention attention matrix is $$Att(X) = Q \times K = {X W}^Q \times ({X W}^K)^T = {X W}^Q (...
user avatar
0 votes
0 answers
16 views

Why is it common to use K=V in Attention layers?

Context: Hi I recently read from the keras docs: "key: Optional key Tensor of shape (B, S, dim). If not given, will use value for both key and value, which is the most common case." I found ...
user avatar
  • 276
0 votes
0 answers
23 views

How to use copy mechanism and attention together?

Is the copy mechanism and attention related for a Neural Machine Translation task when source and target vocabulary are the same? Copy mechanism means unknown words would be copied from source to the ...
user avatar
0 votes
0 answers
22 views

Adding an attention mechanism to variational autoencoder

I'm wondering if it could make sense trying to incorporate an attention mechanism into a variational autoencoder and eventually how to do that. Would it make sense to apply a first layer of processing ...
user avatar
2 votes
1 answer
34 views

Is this aggregation of multiple convolutions of the same input a type of attention or dynamic convolution?

Are there any examples of people performing multiple convolutions at a single depth and then performing feature max aggregation as a convex combination as a form of "dynamic convolutions"? ...
user avatar
  • 165
0 votes
1 answer
32 views

Sensible integer embedding/encoding for distinguishing elements of a set?

I am trying to train a model that takes in a set of feature vectors (which comes with an ID to uniquely identify elements of the set) and outputs a target for each element in the set (in a permutation-...
user avatar
  • 101
2 votes
1 answer
130 views

What is multi-head attention doing mathematically, and how is it different from self-attention?

I'm trying to understand the difference between the concept of self-attention and multi-head attention. The latter is not actually too clear to me. I understand that, in the case of self-attention, we ...
user avatar
0 votes
1 answer
320 views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
user avatar
  • 151
2 votes
1 answer
32 views

Is there a notion of location in Transformer architecture in subsequent self-attention layers?

Transformer architecture (without position embedding) is by the very construction equivariant to the permutation of tokens. Given query $Q \in \mathbb{R}^{n \times d}$ and keys $K \in \mathbb{R}^{n \...
user avatar
1 vote
0 answers
33 views

Why does research on faster Transformers focus on the query-key product?

A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism: $$\text{softmax}\left(\frac{Q K^T}{\sqrt{d}} \right)V,$$ As I understand it, the runtime,...
user avatar
3 votes
2 answers
247 views

In layman terms, what does "attention" do in a transformer?

I heard from many people about the paper titled Attention Is All You Need by Ashish Vaswani et al. What actually does the "attention" do in simple terms? Is it a function, property, or some ...
user avatar
  • 3,099
0 votes
0 answers
28 views

Is the multi-head attention in the transformer a weighted adjacency matrix?

Are multi-head attention matrices weighted adjacency matrices? The job of the multi-head-attention mechanism in transformer models is to determine how likely a word is to appear after another word. In ...
user avatar
1 vote
0 answers
39 views

Are there any benefits of adding attention to linear layers?

Is attention useful only in transformer/convolution layers? Can I add it to linear layers? If yes, how (on a conceptual level, not necessarily the code to implement the layers)?
user avatar
3 votes
0 answers
160 views

Is there a proper initialization technique for the weight matrices in multi-head attention?

Self-attention layers have 4 learnable tensors (in the vanilla formulation): Query matrix $W_Q$ Key matrix $W_K$ Value matrix $W_V$ Output matrix $W_O$ Nice illustration from https://jalammar....
user avatar
2 votes
1 answer
716 views

Why does a transformer not use an activation function following the multi-head attention layer?

I was hoping someone could explain to me why in the transformer model from the "Attention is all you need" paper there is no activation applied after both the multihead attention layer and ...
user avatar
0 votes
0 answers
28 views

How to get Attention Maps from Attention Gates in Attention UNET?

Contex I have Attention UNET for image segmentation. I use it for humans segmentation. Question Everything works fine. I want to get attention maps from my network, so I could see what my UNET is ...
user avatar
1 vote
0 answers
91 views

Couldn't the self-attention mechanism be replaced with a global depth-wise convolution?

The main advantages of the self-attention mechanism are: Ability to capture long-range dependencies Ease to parallelize on GPU or TPU However, I wonder why the same goals cannot be achieved by ...
user avatar
0 votes
1 answer
30 views

What is the bit memory task?

I learned from this post about the so-called bit memory: They froze its self-attention and feed-forward layers and, in separate copies, fine-tuned peripheral layers on each on a wide range of tasks: ...
user avatar
0 votes
0 answers
17 views

Why do we transform feature vectors in attention modules for CNNs

If we have a set of feature maps with dimensions [B, C, H, W] (batch, channel, height, width), why do we transform our feature maps before we calculate their affinity/correlation in attention ...
user avatar
0 votes
1 answer
73 views

Attention mechanism: Why apply multiple different transformations to obtain query, key, value

I have two questions about the structure of attention modules: Since I work with imagery I will be talking about using convolutions on feature maps in order to obtain attention maps. If we have a set ...
user avatar
0 votes
1 answer
120 views

Why people always say the Transformer is parallelizable while the self-attention layer still depends on outputs of all time steps to calculate?

When compared to an RNN seq-to-seq model, people always say the Transformer is parallelizable. In the original Attention Is All You Need paper, it also said that Recurrent models typically factor ...
user avatar
1 vote
1 answer
2k views

Isn't attention mask for BERT model useless?

I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial ...
user avatar
5 votes
1 answer
1k views

In Computer Vision, what is the difference between a transformer and attention?

Having been studying computer vision for a while, I still cannot understand what the difference between a transformer and attention is?
user avatar
  • 53
1 vote
0 answers
151 views

Computing the mean attention distance for ViT

Recently I came across the paper that introduces the Vision Transformer (ViT) "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE". The thing I don't really ...
user avatar
  • 11
2 votes
1 answer
342 views

Are there any advantages of the local attention against convolutions?

Transformer architectures, based on the self-attention mechanism, have achieved outstanding performance in a variety of applications. The main advantage of this approach is that the given token can ...
user avatar
0 votes
0 answers
150 views

How do autoregressive attention mechanism work in multi-headed attention?

[LONG POST!!] I am working on a DNN model that works as an improviser to generate music sequences. The idea of generating music is based on taking a sequence of music nodes (their index representation)...
user avatar
0 votes
0 answers
23 views

Visualizing encoder-attention after ResNet in terms of ResNet input

I have a transform-encoder only architecture, which has the following structure: ...
user avatar
  • 101
1 vote
1 answer
29 views

How is the variational lower bound for hard attention derived in Show, Attend and Tell

How is the jump from line 1 to line 2 done in equation 10 of Show, Attend and Tell? While we're at it, another thing that might be muddying the waters for me is that I'm not clear on what the sum is ...
user avatar
2 votes
0 answers
65 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
user avatar
1 vote
1 answer
41 views

Do RNNs/LSTMs really need to be sequential?

There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly ...
user avatar
  • 13
1 vote
0 answers
40 views

SAGAN - is there a mistake in the original paper?

in the original paper the following scheme of the self-attention appears: https://arxiv.org/pdf/1805.08318.pdf In a later overview: https://arxiv.org/pdf/1906.01529.pdf this scheme appears: ...
user avatar
  • 111
1 vote
1 answer
65 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
user avatar
  • 119
5 votes
2 answers
1k views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
user avatar
1 vote
1 answer
89 views

How is the transformers' output matrix size arrived at?

In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
user avatar
  • 111
1 vote
0 answers
90 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
user avatar
9 votes
3 answers
2k views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
user avatar
1 vote
1 answer
127 views

What is the purpose of "alignment" in the self-attention mechanism of transformers?

I've been reading about transformers & have been having some difficulty understanding the concept of alignment. Based on this article Alignment means matching segments of original text with their ...
user avatar
0 votes
1 answer
124 views

Reasoning behind performance improvement with hopfield networks

In the paper Hopfield networks is all you need, the authors mention that their modern Hopfield network layers are a good replacement for pooling, GRU, LSTM, and attention layers, and tend to ...
user avatar
0 votes
1 answer
48 views

Factors that causing totally different outcomes from an exactly same model and datasets

Here is a model that trains time series data in (batch, step, features) way. I have kept the random state for train test split function the same. Every parameter below the same, running the model ...
user avatar
  • 123
1 vote
0 answers
13 views

How are the parameters $\alpha_i$ of hard attention trained?

I have a question about Show, Attend and Tell: Neural Image CaptionGeneration with Visual Attention paper by Xu. The basic mechanism of stochastic hard attention is that each pixel of the input image ...
user avatar
4 votes
1 answer
824 views

What's the difference between content-based attention and dot-product attention?

I'm following this blog post which enumerates the various types of attention. It mentions content-based attention where the alignment scoring function for the $j$th encoder hidden state with respect ...
user avatar