Skip to main content

Questions tagged [attention]

For questions about the concept of attention in artificial intelligence and machine learning. Attention-like mechanisms were successfully used in natural language processing and computer vision tasks, such as machine translation. For a review of attention-based mechanism used in NLP, take a look at "Attention in Natural Language Processing" by Andrea Galassi et al.

Filter by
Sorted by
Tagged with
0 votes
1 answer
12 views

Attention Layer as input data filter

I can't find decisive sources on that matter. Is it possible to use attention layer as a sort of filter of input data before passing it further to the network? Is it possible to use it to reduce the ...
janek nowaczek's user avatar
1 vote
2 answers
77 views

Can transformer attention make predictions based on analogy?

Suppose I have included 3 examples of an idiosyncratic sentence for training by a transformer: Example 1: Asdfogiug likes Zsdfoiusdhf and Zsdfoiusdhf likes Asdfogiug too. Example 2: Bsodifhas likes ...
James's user avatar
  • 157
0 votes
1 answer
20 views

probability intepreter of attention mechanism in Seq2Seq

Many people explained seq2seq model by explanatory description. However, in my opinion, that is just like a robot who could say something correctly but don't really understand it. Just like the AI did....
tangyao's user avatar
0 votes
1 answer
45 views

Why are the Q and K matrices two separate matrices in attention?

If I understand correctly the attention layer is represented as $$ \begin{align} &softmax(\frac{Q K^T}{\sqrt{d_k}}) V \\ = &softmax(\frac{(s W_q) (s W_k)^T}{\sqrt{d_k}}) V \\ = &softmax(\...
fakedrake's user avatar
  • 103
0 votes
0 answers
28 views

Why can't we use only keys to calculate self-attention?

I was reading about the self-attention mechanism and the paper suggests to have 3 things to be computed: Key, Query and Value. As far as I understood the reason for having Value is to allow ...
Erik Nouroyan's user avatar
0 votes
0 answers
21 views

How is the complexity of the chunked attention computation in "Self Attention Does Not Need O(n2) Memory" independent from the query chunks size?

In Self-attention Does Not Need O(n{2}) Memory the authors present a say to have a constant memory complexity attention algorithm that is sequential in nature and also present an implementation that ...
Daviiid's user avatar
  • 575
0 votes
0 answers
30 views

can Vision transformers be used to retain the relevant features (drop unrelated features from the clutter in image) and map to the specific query

Background, I have good understanding of ML 101 (supervised, unsupervised, tensorflow etc), however just getting into transformers & gen-AI. I have recently started looking into Transformers/ViT ...
cyborgt8's user avatar
1 vote
0 answers
20 views

Can a transformer detect same object with different sizes?

Suppose a vision transformer has trained to detect this cat picture Next we show it another picture of a zoomed-in cat (taken from the same image) and asked it to identify the picture The linearized ...
James's user avatar
  • 157
0 votes
0 answers
18 views

Any LMs that use tanh (generalization) instead of sigmoid within Attention?

Question is in the title. Posts such as this and this mention how this would be possible. I have some colleagues who have anecdotally heard of this being done on very small transformer models but I ...
naston's user avatar
  • 11
2 votes
1 answer
70 views

Can attention models be replaced by non-sigmoid activation functions?

As far as I understand, the attention model in a LLM is used to mitigate the vanishing gradient problem. When using activation functions like the sigmoid function, deep neural networks may lead to ...
A. Darwin's user avatar
  • 143
0 votes
0 answers
24 views

Can the vision transformer solve object permanence?

Suppose a ball is rolling down an incline as follows: In the middle of the path, a curtain blocks the camera view of the ball momentarily. Soon the ball reappears after passing behind the curtain. ...
James's user avatar
  • 157
0 votes
0 answers
17 views

When using local self-attention how do keys and queries wind up having compatible dimensions if if different sized convolutions are used to make them?

In the paper https://arxiv.org/pdf/2112.02143 on page 3 in figure 1 an architecture is described. As can be seen in the upper left, the keys for Local self-attention are generated using a 3x3 ...
Homer Sanchez's user avatar
1 vote
0 answers
40 views

How to Interpret Cross Attention

I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. ...
Kiran Manicka's user avatar
0 votes
0 answers
33 views

why explicit reference of batchsize is required in attention

When I read the code for multi-head attention, I noticed all people consider the batch_size in the forward() For example, the code below was provided by GPT: ...
stayfish's user avatar
0 votes
0 answers
25 views

Is token mask masked in attention of encoders of bert?

I have recently researched on Bert structure. And the paper says we will mask some token at the input in 80%, 10% input be changed and 10% left remained. But I wonder if the mask token in the input be ...
Thành An's user avatar
0 votes
0 answers
22 views

Help understand the output of dot product attention mechanism

I need help interpreting the output of my dot product attention mechanism which I have just started learning. My query vector was $(0,1,0)$, my keys were $key_1 = (0,1,0)$, $key_2 = (1,0,0)$, $key_3 = ...
user1074348's user avatar
0 votes
1 answer
38 views

How to get Complexity per Layer, Sequential Operations and Maximum Path Length in CNN architecture?

In the paper Attention is all you need, here is Table 1, can someone explain what architecture is referred to in the "Convolution" row and hence describe the other 3 columns in it? The other ...
Harry's user avatar
  • 11
0 votes
0 answers
62 views

Why does the algorithm in "Self-attention Does Not Need $O(n^{2})$ Memory" require $O(log n)$ memory when $k, v$ pairs are not ordered?

I am reading Self-attention Does Not Need $O(n^{2})$ Memory which proposes an algorithm that requires $O(1)$ memory for one query and $O(log n)$ memory for self-attention, in theory. In practice the ...
Daviiid's user avatar
  • 575
0 votes
0 answers
35 views

Attention Mechanism: Why don't we just use a simple dot product instead of the Q, K, V matrices?

I am currently learning about Transformers by reading Richard Turner's paper "An Introduction to Transformers". On page 3 of the paper he gave a "naive" approach to build the ...
StockComCat's user avatar
0 votes
2 answers
137 views

How the Q,K,V be calculated in multi-head attention

I want to understand the transformer architecture, so I start with self attention and I understand their mechanism, but when I pass to the multi-head attention I find some difficulties like how ...
LAILA EL OUEDEGHYRY's user avatar
0 votes
0 answers
12 views

Transformers Decoder Inputs (Keys and Values)

I am trying to better understand the encoder-decoder transformer architecture. I understand the high-level intuition behind the concept of keys, queries and values. For example, in the context of CV, ...
user2661372's user avatar
0 votes
0 answers
17 views

Does each head in multihead-attention only get a sub-set of the features?

As far as I've understand, when we have multi-head attention in a layer i.e say 2 heads, then each head gets 1/2 of the features i.e if our embedding size is 512 then each head would get 256 features ...
CutePoison's user avatar
0 votes
1 answer
82 views

How do I code so that the embedding output and input share the same weight matrices?

I am trying to implement the Attention is All You Need paper from scratch. The authors mentioned in section 3.4 that "In our model, we share the same weight matrix between the two embedding ...
OneMoreGamble's user avatar
1 vote
2 answers
146 views

How are sentences turned into a vector in LLM

My understanding of Large Language Models like GPT is that they are special kinds of deep neural networks specifically trained to predict the next word, given the beginning of a sentence. I understand ...
Weier's user avatar
  • 131
1 vote
1 answer
150 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
Kiran Manicka's user avatar
1 vote
3 answers
93 views

Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?

About attention: the Query, Key and Value vectors (before the linear transformations) are just the entire sequence, that is being inputted, or just each token? Chat-GPT nor Youtube didn't give me a ...
CyberLight 64's user avatar
1 vote
0 answers
73 views

SOTA model (PAtt-lite) implementation with Pytorch [closed]

I'm trying to implement a model guided by the paper of PAtt-lite (weights are on https://github.com/jlrex/patt-lite but no implementation provided yet). Using FER2013+ and RAF-DB. The main classes I'...
Robert's user avatar
  • 111
1 vote
2 answers
81 views

Attention with Recurrent Neural Networks

In RNNs, to avoid "forgetting" information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time t on each ...
FluidMechanics Potential Flows's user avatar
0 votes
0 answers
21 views

Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?

here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
Jeremy Kenn's user avatar
1 vote
1 answer
38 views

How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language?

How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language? Since we don't dictate ahead of time what the self-attention layers focus on, how do we ...
Tfovid's user avatar
  • 187
0 votes
1 answer
53 views

Handling Variable Output Token Dimensionality in Transformer Decoders During Inference

I'm curious about something in the decoder part of the Transformers architecture. From what I understand, the Keys and Values come from the output of the encoder part of the Transformer. I understand ...
FluidMechanics Potential Flows's user avatar
0 votes
0 answers
20 views

How to train ViT on smaller datasets?

I know ViTs aren't made for small datasets and low resolution. But have you ever reached traditional CNN accuracy using ViT on CIFAR10/100. I have been playing around with ViT on CIFAR10 and 100. But ...
v1998199904's user avatar
1 vote
1 answer
73 views

Is the multi-headed projection matrix in self-attention redundant?

As I understand it, the forward pass for a transformer model looks as follows: x += self_attention(x) x = layernorm(x) x += ffn(x) Breaking that down a bit (excuse ...
Sue Doh Nimh's user avatar
0 votes
0 answers
31 views

Attention module (CBAM) in CNN tend to saturate values to 1

In the context of image classification, I am using a feature extractor based on a resnet-like architecture (ResNet12): four residual blocks, each of which is made of two consecutive conv3x3, batch ...
Lorenzo's user avatar
0 votes
1 answer
63 views

how can I interpret attention weights matrix? Are they reliable?

I've fine-tuned two different models (Bert and Roberta) on a dataset for a binary classification task and I'm comparing the sentences where the models predict wrong. I decided to use attention weights ...
Shayan's user avatar
  • 21
0 votes
0 answers
32 views

Are K and V values reused in each decoder layer's cross-Attention in the original "Attention is all you need" paper?

I'm working with Transformers and have a question about the encoder-decoder structure. In each decoder layer's cross-attention, are the K and V pairs from the corresponding encoder layer reused for ...
Dennis Yang's user avatar
0 votes
0 answers
31 views

Why are rows of Attention Weights in a Hopfield Transformer the same?

I'm working on building a Hopfield Transformer using the github code from the paper (https://github.com/ml-jku/hopfield-layers/tree/master/hflayers) to forecast a timeseries dataset with 48 variables, ...
Ryan Bose-Roy's user avatar
0 votes
0 answers
51 views

Concatenating the positional encoding

As per "Attention is all you need" etc., positional encoding is added to the embedded word vector input. My knee-jerk reaction is that this would muddle the "signal" of the word ...
SuaveSouris's user avatar
0 votes
0 answers
37 views

What am I doing wrong that result in a graph indicating better gradients in non-scaled dot-product attention compared to the scaled version?

I'm trying to visualize how the gradients change as we're increasing $d_{k}$ in the scaled dot-product attention and compare it to its non scaled version but I'm failing to produce a reasonable graph ...
Daviiid's user avatar
  • 575
3 votes
1 answer
188 views

Understanding the function of attention layers in a convolutional neural network (U-Net in a diffusion model)

I am trying to understand the neural network architecture used by Ho et al. in "Denoising Diffusion Probabilistic Models" (paper, source code). They include self-attention layers in the ...
Rational Function's user avatar
0 votes
0 answers
49 views

Do I need residual block in a transformer model if vanishing gradients don't exist?

In this example l’Afrique is x^3 and the attention is being computed for this word A^3(l’Afrique). In the image above Andrew Ng indicates that the word with the biggest wieght in the computation of A^...
Stef's user avatar
  • 95
0 votes
2 answers
106 views

Transformers - how do the decoder attention input matrices look like, in terms of future tokens?

I have a question regarding the original transformer implementation (as in "Attention is all you need"). Assuming I want to translate English to German. In the Decoder part, in the self-attention ...
Ben Lahav's user avatar
-1 votes
1 answer
52 views

add a layer in deep learning model pytorch [closed]

I have a deep learning model, and I want to add a attention layer in pytorch. in the forward function, I add the attention as this: however, when I check the weights of the attention, all of the ...
Sadcow's user avatar
  • 99
0 votes
0 answers
27 views

Could we use another attention module on the outputs of the attention heads in the transformer architecture?

Before concatenating the heads in MHAtt, could we add another attention module with the heads as input to combine them? Thus, for each head we would get a value matrice enriched with the outputs of ...
d3nigma's user avatar
  • 23
1 vote
1 answer
768 views

Is Softmax Necessary as the Activation Function for Self-Attention Mechanisms?

I’m curious about the mathematical reasoning behind the use of the softmax function as the activation function in self-attention mechanisms within neural networks. Specifically, I’m interested in ...
Kasia's user avatar
  • 303
0 votes
1 answer
109 views

Why is it called multi-headed attention?

Why do we call the attention layer in transformers multi-headed attention when in practice all the attention matrices from different heads (W,K,V) for a single layer are concatenated to perform the ...
Tarique's user avatar
  • 103
0 votes
0 answers
15 views

Are the outputs of layers in Attention Is All You Need interpretable by mapping to tokens?

In the basic transformer model from 2017, I'm a bit confused what the outputs of each layer are supposed to be. Are they embeddings? If so, does that mean you could examine a given output from a given ...
Stan Shunpike's user avatar
0 votes
2 answers
172 views

Why encoders are required in Transformers

In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....). I mean even a decoder also has the same ...
Swastik's user avatar
  • 101
0 votes
0 answers
25 views

What algorithms could I use if I want to increase the accuracy of matched keypoints in an image pair?

Let's say that I used a keypoint detector like SIFT or SuperPoint to detect keypoints in image 1 and 2. Afterwards, I used a keypoint matcher to match corresponding keypoints in this image pair. The ...
Surzilla's user avatar
  • 101
3 votes
1 answer
840 views

Aren't context lengths for transformers an artificial restriction?

Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference. : https://i....
Robert Wegner's user avatar