# Tag Info

0

An RNN processes words one by one. For example on the sentence "man eats dog", it will: Fully process "man", producing an output $y_1$ and hidden units $h_1$. Fully process "eats", now using also the previous output and/or hidden units. Finally process "dog", again using the previous output and/or hidden units $y_2$ ...

2

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-attention operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other ...

1

It is true that when using local attention with a window of size 5, the "receptive field" is the same as a CNN with kernel size 5 (or two CNN layers with kernel size 3). However, there is a key difference in how the learned weights are applied to the inputs. In a CNN, the values of the many convolutional kernels are learned, but once learned, the ...

Top 50 recent answers are included