Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

Question

Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this with an example?

Answer 1

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the length of the input sequence.

What you are getting confused with is that RNN's can make predictions relating to sequences, that is imagine now rather than one $x$ we have a sequence of related, i.e. not i.i.d. data such as time series data, data $\{x_i\}_{i=1}^n$. Assuming we are given some initial $h_0$ (a hidden state) then an RNN will take as input $x_1$ and $h_0$ and output a prediction $y_1$ and a new hidden state $h_1$. In general an RNN will take as input $x_n$ and $h_{n-1}$ and output $y_n$ and $h_n$ where the hidden state is passed as input to the RNN at the next time step. However, the dimensionality of all the $x$'s and $h$'s will all be the same, i.e. $x_i \in \mathbb{R}^d$ and $h_i \in \mathbb{R}^c$ for all $i$, where $d$ does not necessarily have to be equal to $c$ (and in my experience rarely is).

Note that RNN's can also perform sequence to sequence prediction (such as language translation) where the predicted sequence can be a different length to the input sequence, as is the case when doing translation (the input sentence is not necessarily the same length in the translated language). They do this by having an encoder and a decoder which are two separate RNNs. The encoder is fed the input sequence and we maintain all the hidden states outputted by the encoder $\{e_i\}_{i=1}^n$. The decoder is then given a token that represents the start of a sequence and the last hidden state of the encoder $e_n$ as input to which it will make a prediction on what the word should be (I believe but am not 100% certain that the decoder outputs a probability distribution over the dictionary of words it can predict from) and then the word chosen is passed as input to the decoder at the next step, along with the hidden state from the decoder at the previous step and another word is predicted. This continues until the system predicts an End of Sequence token (or until it is forced to by some time limit). More details can be found in this paper but I believe in NLP it is much more common to use attention models now rather than the methods introduced in this paper.

Now, with this data $\{x_i\}_{i=1}^n$ you could in theory pass each one individually to a FFNN but a) it would not capture the sequential nature of the data as the assumption is that each data point is independent of the others and you can see this from the architecture of a FFNN -- they are Directed Acyclic Graphs, the Acyclic-ness is what causes the issue as there is no recursion which stops any sequential information from being passed from one time step to another and b) training using SGD would likely cause issues as we have violated the i.i.d. assumption needed for SGD to converge to local optima. More info on the i.i.d. requirement can be found here.