What is a recurrent neural network?

Question

Surprisingly, this wasn't asked before - at least I didn't find anything besides some vaguely related questions.

So, what is a recurrent neural network, and what are their advantages over regular (or feed-forward) neural networks?

Answer 1

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information.

Difference with traditional Neural networks using pictures from this book:

And, an RNN:

Notice the difference -- feedforward neural networks' connections do not form cycles. If we relax this condition, and allow cyclical connections as well, we obtain recurrent neural networks (RNNs). You can see that in the hidden layer of the architecture.

While the difference between a multilayer perceptron and an RNN may seem trivial, the implications for sequence learning are far-reaching. An MLP can only map from input to output vectors, whereas an RNN can in principle map from the entire history of previous inputs to each output. Indeed, the equivalent result to the universal approximation theory for MLPs is that an RNN with a sufficient number of hidden units can approximate any measurable sequence-to-sequence mapping to arbitrary accuracy.

Important takeaway:

The recurrent connections allow a 'memory' of previous inputs to persist in the network's internal state, and thereby influence the network output.

Talking in terms of advantages is not appropriate as they both are state-of-the-art and are particularly good at certain tasks. A broad category of tasks that RNN excel at is:

Sequence Labelling

The goal of sequence labelling is to assign sequences of labels, drawn from a fixed alphabet, to sequences of input data.

Ex: Transcribe a sequence of acoustic features with spoken words (speech recognition), or a sequence of video frames with hand gestures (gesture recognition).

Some of the sub-tasks in sequence labelling are:

Sequence Classification

Label sequences are constrained to be of length one. This is referred to as sequence classification, since each input sequence is assigned to a single class. Examples of sequence classification task include the identification of a single spoken work and the recognition of an individual handwritten letter.

Segment Classification

Segment classification refers to those tasks where the target sequences consist of multiple labels, but the locations of the labels -- that is, the positions of the input segments to which the labels apply -- are known in advance.

Answer 2

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks.

An RNN can be trained using back-propagation through time (BBTT), such that these backward or self-connections "memorise" previously seen inputs. Hence, these connections are mainly used to track temporal relations between elements of a sequence of inputs, which makes RNNs well suited to sequence prediction and similar tasks.

There are several RNN models: for example, RNNs with LSTM or GRU units. LSTM (or GRU) is an RNN whose single units perform a more complex transformation than a unit in a "plain RNN", which performs a linear transformation of the input followed by the application of a non-linear function (e.g. ReLU) to this linear transformation. In theory, "plain RNN" are as powerful as RNNs with LSTM units. In practice, they suffer from the "vanishing and exploding gradients" problem. Hence, in practice, LSTMs (or similar sophisticated recurrent units) are used.

Answer 3

Recurrent Neural Network(RNN):

• can remember the output of the previous step to use it in the next step.
• is used for NLP(Natural Language Processing). *NLP is the technology which enables computers to understand and communicate with human language.
• is RNN() in PyTorch.

In addition, there is LSTM(Long Short-Term Memory) which is the improved version of RNN.

LSTM(Long Short-Term Memory):

• can remember the output of the previous steps to use them in the next step while RNN can remember the output of only the previous step.
• has more ability than Recurrent Layer(RNN).
• is slower to train data than RNN because the computation is more complex.
• is used for NLP.
• is LSTM() in PyTorch.