8
$\begingroup$

If recurrent neural networks (RNNs) are used to capture prior information, couldn't the same thing be achieved by a feedforward neural network (FFNN) or multi-layer perceptron (MLP) where the inputs are ordered sequentially?

Here's an example I saw where the top line of each section represents letters typed and the next row represents the predicted next character (red letters in the next row means a confident prediction).

enter image description here

Wouldn't it be simpler to just pass the $X$ number of letters leading up to the last letter into an FFNN?

For example, if $X$ equaled 4, the following might be the input to the FFNN

S, T, A, C => Prediction: K
$\endgroup$
2
$\begingroup$

Assumptions

Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others.

For example, your proposed structure of "just pass the $X$ number of letters leading up to the last letter into an FFNN" makes an assumption that all the information relevant for the decision is fully obtainable from the $X$ previous letters, and $(X+1)$st and earlier input letters are not relevant - in some sense, an extension of the Markov property. Obviously, that's not true in many cases, there are all kinds of structures where long term relationships matter, and assuming that they don't lead to a model that intentionally doesn't take such relationships into account. Furthermore, it would make an independence assumption that the effect of $X$th, $(X-1)$st and $(X-2)$nd elements on the current output is entirely distinct and separate, you don't make an assumption that those features are related, while in most real problems they are.

The classic RNN structures also make some implicit assumptions, namely, that only the preceding elements are relevant for the decision (which is wrong for some problems, where information from the following items is also required), and that the transformative relationship between the input, output and the passed-on state is the same for all elements in the chain, and that it doesn't change over time; That's also not certainly true in all cases, this is quite a strong restriction, but that's generally less wrong than the assumption that the last $X$ elements are sufficient, and powerful true (or mostly true) restrictions are useful (e.g. the No Free Lunch Theorem applies) for models that generalize better; just like e.g. enforcing translational invariance for image analysis models, etc.

$\endgroup$
  • $\begingroup$ This is a very clear and thorough answer to the question being asked. Thanks. $\endgroup$ – SuperCodeBrah Nov 3 '19 at 18:25
7
$\begingroup$

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only through gradient descent. RNN or LSTM however have "time" as a mechanism built into the model. The model loops through the model sequentially and have a real "sense of time" even before the model is trained. The model also have "memory" of previous data points to help the prediction. The architecture is based on the progress of time and the gradient are propagated through time as well. This is a much more intuitive way to process time-series data.

A 1D CNN also will work for the task. An example of CNN in time series data is wavenet, which uses CNN for generating incredibly life like speech using dilated convolution neural network. For whether LSTM or CNN works better, it depends on the data. You should try experimenting with both networks to see which works best.

Suppose you need to classify a video's genre. It is much simpler to watch it in sequence then seeing frames of it playing randomly in front of your eyes. This is why an RNN or an LSTM works better in time series data.

$\endgroup$
  • 2
    $\begingroup$ RNNs make sense to humans, but what does the network really "know"? Wouldn't each layer of the network treat the previous layer's output as just another input? I could see there being a flexibility to LSTMs in that the designer doesn't need to specify a number of time periods. $\endgroup$ – SuperCodeBrah Nov 2 '19 at 15:20
  • 3
    $\begingroup$ The RNN or LSTM treat the hidden units as one of the input to the network. It helps gradients to propagate through time and also the processed features of the past neurons to be treated as an input. The network work through time data sequentially which helps the data understand the data easily and no need for the network to learn the concept of time. $\endgroup$ – Clement Hui Nov 2 '19 at 15:23
  • 1
    $\begingroup$ Okay, I think I see. So the RNN builds a relationship among inputs and not just between inputs and output(s). Could a 1D CNN also replicate this? $\endgroup$ – SuperCodeBrah Nov 2 '19 at 15:39
  • 1
    $\begingroup$ 1D CNN have been proven to work quite good for time series data as well. For example, wavenet, a voice generation network, also uses CNN as the network. However CNN and LSTM works well in the task, but experimenting is the way to find out which is better $\endgroup$ – Clement Hui Nov 3 '19 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.