Surprisingly this wasn't asked before - at least I didn't find anything besides some vaguely related questions.
So, what is a recurrent neural network, and what are their advantages over regular NNs?
Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.Sign up to join this community
Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information.
Difference with traditional Neural networks using pictures from this book:
And, an RNN:
Notice the difference -- feedforward neural networks' connections do not form cycles. If we relax this condition, and allow cyclical connections as well, we obtain recurrent neural networks (RNNs). You can see that in the hidden layer of the architecture.
While the difference between a multilayer perceptron and an RNN may seem trivial, the implications for sequence learning are far-reaching. An MLP can only map from input to output vectors, whereas an RNN can in principle map from the entire history of previous inputs to each output. Indeed, the equivalent result to the universal approximation theory for MLPs is that an RNN with a sufficient number of hidden units can approximate any measurable sequence-to-sequence mapping to arbitrary accuracy.
The recurrent connections allow a 'memory' of previous inputs to persist in the network's internal state, and thereby influence the network output.
Talking in terms of advantages is not appropriate as they both are state-of-the-art and are particularly good at certain tasks. A broad category of tasks that RNN excel at is:
The goal of sequence labelling is to assign sequences of labels, drawn from a fixed alphabet, to sequences of input data.
Ex: Transcribe a sequence of acoustic features with spoken words (speech recognition), or a sequence of video frames with hand gestures (gesture recognition).
Some of the sub-tasks in sequence labelling are:
Label sequences are constrained to be of length one. This is referred to as sequence classification, since each input sequence is assigned to a single class. Examples of sequence classification task include the identification of a single spoken work and the recognition of an individual handwritten letter.
Segment classification refers to those tasks where the target sequences consist of multiple labels, but the locations of the labels -- that is, the positions of the input segments to which the labels apply -- are known in advance.
A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks.
An RNN can be trained using back-propagation through time (BBTT), such that these backward or self-connections "memorise" previously seen inputs. Hence, these connections are mainly used to track temporal relations between elements of a sequence of inputs, which makes RNNs well suited to sequence prediction and similar tasks.
There are several RNN models: for example, RNNs with LSTM or GRU units. LSTM (or GRU) is an RNN whose single units perform a more complex transformation than a unit in a "plain RNN", which performs a linear transformation of the input followed by the application of a non-linear function (e.g. ReLU) to this linear transformation. In theory, "plain RNN" are as powerful as RNNs with LSTM units. In practice, they suffer from the "vanishing and exploding gradients" problem. Hence, in practice, LSTMs (or similar sophisticated recurrent units) are used.