New answers tagged attention
0
An RNN processes words one by one. For example on the sentence "man eats dog", it will:
Fully process "man", producing an output $y_1$ and hidden units $h_1$.
Fully process "eats", now using also the previous output and/or hidden units.
Finally process "dog", again using the previous output and/or hidden units $y_2$ ...
2
The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-attention operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other ...
1
It is true that when using local attention with a window of size 5, the "receptive field" is the same as a CNN with kernel size 5 (or two CNN layers with kernel size 3). However, there is a key difference in how the learned weights are applied to the inputs.
In a CNN, the values of the many convolutional kernels are learned, but once learned, the ...
Top 50 recent answers are included
