Questions tagged [attention]

For questions related to attention based AI approaches, including gating at the network cell level, selection of models within a model container, virtual pan and zoom within an image field, selection of FFT window size based on expected audio frequency range, setting of criteria to trigger modification of processing priorities, and other mechanisms for changing the focus of attention.

Filter by
Sorted by
Tagged with
2
votes
0answers
20 views

What is the difference between Squeeze-and-excite and bottleneck modules from Mobilenet v2?

Squezee-and-excite networks introduced SE blocks, while MobileNet v2 introduced linear bottlenecks. What is the effective difference between these two concepts? Is it only implementation (depth-wise ...
2
votes
1answer
33 views

How do I tag the most interesting parts of a video?

This is a follow-up question from my previous question here. I'm new to ML/DL, and one thing I need to do is to use a machine or deep learning video attention model which as the name suggests, can tag ...
0
votes
0answers
31 views

Understanding CNN+LSTM concept with attention and need help

I have a question about the context of CNN and LSTM. I have trained a CNN network for image classification. However, I would like to combine it with LSTM for visualizing the attention weights. So, I ...
2
votes
0answers
30 views

Does GraphSage use hard attention?

I was reading the recent paper Graph Representation Learning via Hard and Channel-Wise Attention Networks, where the authors claim that there is no hard attention operator for graph data. From my ...
4
votes
0answers
83 views

What is the difference between GAT and GaAN?

I was looking at two papers Graph Attention Networks (GAT) by Petar Veličković and GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs by Jiani Zhang. I'm trying to ...
3
votes
1answer
198 views

A mathematical explanation of Attention Mechanism

I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in ...
2
votes
1answer
261 views

Why is dot product attention faster than additive attention?

In section 3.2.1 of Attention Is All You Need the claim is made that: Dot-product attention is identical to our algorithm, except for the scaling factor of $\frac{1}{\sqrt{d_k}}$. Additive ...
0
votes
0answers
8 views

Emergence and attention in non-approximated deep neural networks?

I am reading thesis https://tel.archives-ouvertes.fr/tel-00850289v2 about use of mean field theory for the stochastic approximation of neural networks. There are lot of such research in arxiv cond-nn ...
1
vote
0answers
68 views

BERT intermediate layer utility

I'm looking for the BERT model you can find here. As I look to the attention mechanism, I don't understand why in the BERT encoder we have an intermediate layer between the attention and neural ...
2
votes
0answers
76 views

How do the Sine and Cosine functions encode position in the “Attention is All You Need” paper?

After going through both the "Illustrated Transformer" and "Annotated Transformer" blog posts, I still don't understand how the sinusoidal encodings are representing the position of elements in the ...
3
votes
0answers
94 views

What is the intuition behind the calculation of the similarity between encoder and decoder states?

Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y_1, y_2, \dots, y_J$ from $x_1, x_2, \dots x_I$: $$P(...
1
vote
2answers
234 views

How to use RNN With Attention Mechanism on Non Textual Data?

Recurrent Neural Networks (RNN) With Attention Mechanism is generally used for Machine Translation and Natural Language Processing. In Python, implementation of RNN With Attention Mechanism is ...
4
votes
1answer
687 views

Will attention based networks prevail over RNN and LSTM?

There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...