I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in recurrent networks), should eventually lead to the network focusing on certain parts of the input/source. So what is attention mechanism really adding to the network?
A potential answer in the case of Encoder-Decoder RNNs is:
The most important distinguishing feature of this approach from the basic encoder–decoder is that it does not attempt to encode a whole input sentence into a single fixed-length vector. Instead, it encodes the input sentence into a sequence of vectors and chooses a subset of these vectors adaptively while decoding the translation. This frees a neural translation model from having to squash all the information of a source sentence, regardless of its length, into a fixed-length vector. We show this allows a model to cope better with long sentences.
- Neural Machine Translation by Jointly Learning to Align and Translate
which made sense and the paper says that it worked better for NMT.
A previous study indicated that breaking down the sentence into phrases could lead to better results:
In this paper, we propose a way to address this issue by automatically segmenting an input sentence into phrases that can be easily translated by the neural network translation model. Once each segment has been independently translated by the neural machine translation model, the translated clauses are concatenated to form a final translation. Empirical results show a significant improvement in translation quality for long sentences.
- Overcoming the Curse of Sentence Length for Neural Machine Translation using Automatic Segmentation
which paved the way for further research resulting in attention models.
I was also going through an article on Attention is not quite all you need where it said something similar:
An LSTM has to learn to sequentially retain past values together in a single internal state across multiple RNN iterations, whereas attention can recall past sequence values at any point in a single forward pass.
and a more curated blog on the family of attention mechanism gives insight on how different ways have been formulated for implementing the concept: Attention? Attention!
Specifically, I want to know how attention mechanism is formulated for this task (aforementioned) or in general. A detailed mathematical insight would be helpful, probably somewhat on these lines: Understanding Attention in NN mathematically