We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [attention]

For questions related to attention based AI approaches, including gating at the network cell level, selection of models within a model container, virtual pan and zoom within an image field, selection of FFT window size based on expected audio frequency range, setting of criteria to trigger modification of processing priorities, and other mechanisms for changing the focus of attention.

Filter by
Sorted by
Tagged with
0
votes
1answer
18 views

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether ...
2
votes
0answers
35 views

Why is the transformer for time series forecasting faster than RNN?

I've been reading different papers which implements the Transformer for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. ...
2
votes
0answers
22 views

What is the difference between Squeeze-and-excite and bottleneck modules from Mobilenet v2?

Squezee-and-excite networks introduced SE blocks, while MobileNet v2 introduced linear bottlenecks. What is the effective difference between these two concepts? Is it only implementation (depth-wise ...
2
votes
1answer
33 views

How do I tag the most interesting parts of a video?

This is a follow-up question from my previous question here. I'm new to ML/DL, and one thing I need to do is to use a machine or deep learning video attention model which as the name suggests, can tag ...
1
vote
0answers
35 views

Understanding CNN+LSTM concept with attention and need help

I have a question about the context of CNN and LSTM. I have trained a CNN network for image classification. However, I would like to combine it with LSTM for visualizing the attention weights. So, I ...
2
votes
0answers
34 views

Does GraphSage use hard attention?

I was reading the recent paper Graph Representation Learning via Hard and Channel-Wise Attention Networks, where the authors claim that there is no hard attention operator for graph data. From my ...
4
votes
0answers
110 views

What is the difference between GAT and GaAN?

I was looking at two papers Graph Attention Networks (GAT) by Petar Veličković and GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs by Jiani Zhang. I'm trying to ...
3
votes
1answer
257 views

A mathematical explanation of Attention Mechanism

I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in ...
0
votes
0answers
36 views

Why don't people use nonlinear activation functions after projecting the query key value in attention?

Why don't people use nonlinear activation functions after projecting the query key value in attention? It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing ...
2
votes
1answer
344 views

Why is dot product attention faster than additive attention?

In section 3.2.1 of Attention Is All You Need the claim is made that: Dot-product attention is identical to our algorithm, except for the scaling factor of $\frac{1}{\sqrt{d_k}}$. Additive ...
0
votes
0answers
8 views

Emergence and attention in non-approximated deep neural networks?

I am reading thesis https://tel.archives-ouvertes.fr/tel-00850289v2 about use of mean field theory for the stochastic approximation of neural networks. There are lot of such research in arxiv cond-nn ...
2
votes
0answers
80 views

Why does the BERT encoder have an intermediate layer between the attention and neural network layers with a bigger output?

I am reading the BERT paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. As I look at the attention mechanism, I don't understand why in the BERT encoder we have ...
2
votes
0answers
94 views

How do the sine and cosine functions encode position in the transformer?

After going through both the "Illustrated Transformer" and "Annotated Transformer" blog posts, I still don't understand how the sinusoidal encodings are representing the position of elements in the ...
3
votes
0answers
94 views

What is the intuition behind the calculation of the similarity between encoder and decoder states?

Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y_1, y_2, \dots, y_J$ from $x_1, x_2, \dots x_I$: $$P(...
1
vote
2answers
248 views

How to use RNN With Attention Mechanism on Non Textual Data?

Recurrent Neural Networks (RNN) With Attention Mechanism is generally used for Machine Translation and Natural Language Processing. In Python, implementation of RNN With Attention Mechanism is ...
4
votes
1answer
757 views

Will attention based networks prevail over RNN and LSTM?

There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...
12
votes
2answers
155 views

Is there any artificial intelligence that possesses “concentration”?

Humans can do multiple tasks at the same (e.g. reading while listening to music), but we memorize information from less focused sources with worse efficiency than we do from our main focus or task. ...