Questions tagged [attention]

For questions about the concept of attention in artificial intelligence and machine learning. Attention-like mechanisms were successfully used in natural language processing and computer vision tasks, such as machine translation. For a review of attention-based mechanism used in NLP, take a look at "Attention in Natural Language Processing" by Andrea Galassi et al.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

LXMERT huggingface [closed]

I'm trying to use LXMERT multimodal model for facebook's hateful memes challenge. For using LXMERT I need to pass visual_feats, visual_pos arguments, so I have extracted features using Faster R-CNN ...
2
votes
0answers
12 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
1
vote
1answer
31 views

Do RNNs/LSTMs really need to be sequential?

There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly ...
1
vote
0answers
34 views

SAGAN - is there a mistake in the original paper?

in the original paper the following scheme of the self-attention appears: https://arxiv.org/pdf/1805.08318.pdf In a later overview: https://arxiv.org/pdf/1906.01529.pdf this scheme appears: ...
0
votes
1answer
34 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
1
vote
1answer
45 views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
1
vote
1answer
43 views

How is the transformers' output matrix size arrived at?

In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
1
vote
0answers
27 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
1
vote
2answers
54 views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
0
votes
0answers
23 views

What is the purpose of “alignment” in the self-attention mechanism of transformers?

I've been reading about transformers & have been having some difficulty understanding the concept of alignment. Based on this article Alignment means matching segments of original text with their ...
0
votes
1answer
54 views

Reasoning behind performance improvement with hopfield networks

In the paper Hopfield networks is all you need, the authors mention that their modern Hopfield network layers are a good replacement for pooling, GRU, LSTM, and attention layers, and tend to ...
0
votes
1answer
46 views

Factors that causing totally different outcomes from an exactly same model and datasets

Here is a model that trains time series data in (batch, step, features) way. I have kept the random state for train test split function the same. Every parameter below the same, running the model ...
0
votes
0answers
23 views

Is the dimensionality of the query and key tensors in this implementation of the transformer's self-attention correct?

Looking at this tutorial matmul_qk = tf.matmul(q, k, transpose_b=True) # (..., seq_len_q, seq_len_k) is the output of two tensors multiplied of shape: ...
1
vote
0answers
12 views

How are the parameters $\alpha_i$ of hard attention trained?

I have a question about Show, Attend and Tell: Neural Image CaptionGeneration with Visual Attention paper by Xu. The basic mechanism of stochastic hard attention is that each pixel of the input image ...
2
votes
1answer
88 views

What's the difference between content-based attention and dot-product attention?

I'm following this blog post which enumerates the various types of attention. It mentions content-based attention where the alignment scoring function for the $j$th encoder hidden state with respect ...
1
vote
2answers
87 views

Recent deep learning textbook (i.e. covering at least GANs, LSTM and transformers and attention)

I am searching for an academic (i.e. with maths formulae) textbook which covers (at least) the following: GAN LSTM and transformers (e.g. seq2seq) Attention mechanism The closest match I got is Deep ...
0
votes
1answer
33 views

Can the attention mechanism improve the performance in the case of short sequences?

I am aware that the attention mechanism can be used to deal with long sequences, where problems related to gradient vanishing and, more generally, representing effectively the whole sequence arise. ...
1
vote
1answer
33 views

In attention models with multiple layers, are weight matrices shared across layers?

In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers? Consider as an example, "Attention is ...
0
votes
2answers
81 views

In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?

In the Attention is all you need paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture $$ \text { Attention }(Q, K, V)=\operatorname{...
2
votes
0answers
72 views

How to handle long sequences with transformers?

I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during ...
3
votes
2answers
93 views

What is different in each head of a multi-head attention mechanism?

I have a difficult time understanding the "multi-head" notion in the original transformer paper. What makes the learning in each head unique? Why doesn't the neural network learn the same ...
0
votes
1answer
121 views

What is the difference between Attention Gate and CNN filters?

Attention models/gates are used to focus/pay attention to the important regions. According to this paper, the authors describe that a model with Attention Gate (AG) can be trained from scratch. Then ...
1
vote
0answers
99 views

What is the gradient of an attention unit?

The paper Attention Is All You Need describes the Transformer architecture, which describes attention as a function of the queries $Q = x W^Q$, keys $K = x W^K$, and values $V = x W^V$: $\text{...
3
votes
1answer
65 views

What is the cost function of a transformer?

The paper Attention Is All You Need describes the transformer architecture that has an encoder and a decoder. However, I wasn't clear on what the cost function to minimize is for such an architecture. ...
1
vote
1answer
26 views

Is the Decoder mask (triangular mask) applied only in the first decoder block, or to all blocks in Decoder?

The Decoder mask, also called "look-ahead mask", is applied in the Decoder side to prevent it from attending future tokens. Something like this: ...
2
votes
1answer
41 views

Transformers: how to get the output (keys and values) of the encoder?

I was reading the paper Attention Is All You Need. It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ ...
2
votes
1answer
32 views

Transformers: how does the decoder final layer output the desired token?

In the paper Attention Is All You Need, this section confuses me: In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
0
votes
0answers
10 views

Best strategy for Classification of Science Subjects. Phy, Chem , Maths and Bio? BERT, Transformers, Attention+SLTM, Self-Attention+LSTM?

I am working on a project where I have to first classify the Subjects of the given question and then the respective Chapter and then the sub-topic. In a nutshell, I have to predict the Subject, Grade ...
4
votes
1answer
79 views

Why are “Transformers” called this way?

What is the reason behind the name "Transformers", for Multi Head Self-Attention-based neural networks from Attention is All You Need? I have been googling this question for a long time, and ...
2
votes
1answer
60 views

Any comparison between transformer and RNN+Attention on the same dataset?

I am wondering what is believed to be the reason for superiority of transformer? I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. ...
5
votes
3answers
300 views

What is the purpose of Decoder mask (triangular mask) in Transformer?

I'm trying to implement transformer model using this tutorial. In the decoder block of the Transformer model, a mask is passed to "pad and mask future tokens in the input received by the decoder&...
1
vote
0answers
55 views

How are weight matrices in attention trained?

I have been looking into transformers lately and been reading tons of tutorials. All of them address the intuition behind attention, which I understand, but they treat training the weight matrices for ...
1
vote
0answers
35 views

Is it possible to express attention as a Fourier convolution?

Convolutions can be expressed as a matrix-multiplication (see e.g. this post) and as an element-wise multiplication using the Fourier domain (https://en.wikipedia.org/wiki/Convolution_theorem). ...
1
vote
1answer
28 views

Appropriate metric and approach for natural language generation for small sentences

I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 ...
1
vote
0answers
26 views

How do non-local neural networks relate to attention and self-attention?

I've been reading non-local neural networks as explained in the original paper. My understanding is that they solve the restrained reception of local filters. I see how they are different from ...
2
votes
1answer
210 views

What is the weight matrix in self-attention?

I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention ...
5
votes
1answer
1k views

What exactly are the “parameters” in GPT-3's 175 billion parameters and how are they chosen/generated?

When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that ...
0
votes
0answers
39 views

Where can I place an attention module for different computer vision tasks?

Where can I place an attention module for different computer vision tasks? For example, in this Github issue, the author is not sure where to place the attention module. Are there any rules or best ...
0
votes
1answer
86 views

What is the big fuzz about SHA-RNN versus Transformers?

In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because ...
0
votes
0answers
23 views

Which model structure is best suited to build a Math OCR (img2latex)? RAM, DRAM, CRNN or Attention OCR?

I am trying to build an OCR which can read the Mathematical equations just like MAthPix and im2markup. im2markup by HarvardNLP seems like a good model but the thing is that it is built using Torch and ...
0
votes
1answer
176 views

Transformers - is the self attention matrix softmax output (layer 1) symmetric?

Let's assume, that we embedded a vector of length 49 into a matrix using 512-d embeddings. If we then multiply the matrix by it transposed version we receive a matrix of 49 by 49. Which is symmetric. ...
3
votes
1answer
161 views

What are the keys and values of the attention model for the encoder and decoder in the “Attention Is All You Need” paper?

I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and ...
2
votes
0answers
44 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
4
votes
1answer
409 views

What is the intuition behind the attention mechanism?

Attention idea is one of the most influential ideas in deep learning. The main idea behind attention technique is that it allows the decoder to "look back” at the complete input and extracts ...
1
vote
2answers
306 views

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this constrains the ...
2
votes
0answers
27 views

How is visual attention mechanism different from a two branch convolutional neural network?

I am doing some research on the visual attention mechanism in remote sensing domain (where the features learnt from one layer are highlighted using the attention mask derived from another layer). From ...
1
vote
0answers
23 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
7
votes
1answer
576 views

What is the intuition behind the dot product attention?

I am watching Attention all you need, In that what is the intuition behind the dot product attention? $$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$ becomes: $$A(Q,K, V) = softmax(QK^...
2
votes
0answers
47 views

How does positional encoding work in the transformer model?

In the transformer model, to incorporate positional information of texts, the researchers have added a positional encoding to the model. How does positional encoding work? How does the positional ...
2
votes
0answers
29 views

Impact or applications of introducing attention in deep networks modelling multi-agent systems

I have been reading quite a lot about the research progress in the domain of self attention-based neural networks that were introduced by Google Inc. in their paper titled "Attention is all you ...