1
$\begingroup$

Understandably RNNs are very good at solving problems involving audio, video and text processing due to the arbitrary input's length of this sort of data.

What I don't understand is why RNNs are also superior at predicting time series data and why we use them over simple MLP DNNs.

Say I wanted to predict what the value in the time series is at $t+1$. I would take a window of, let's say, $t-50, t-49, \dots, t$, and then feed loads of sampled training data into a network. I could either choose to have a single LSTM unit remembering the entire window and basing the predictions on that, or I could simply make a 50 neuron wide MLP network.

What exactly is it about RNNs that makes them better in this scenario or any scenario for that matter?

I understand that the LSTM would have substantially fewer weights (in this scenario) and should be less computationally intensive, but, apart from that, I don't see any difference in these two methods.

$\endgroup$

3 Answers 3

1
$\begingroup$

In an RNN, the output of the previous state is passed as an input to the current state. Intuitively, there is a temporal (time-based) relationship in the way in which input is processed in an RNN. It can understand how the current state was achieved on the basis of the previous values, i.e value at time-step $t$ is a result of value at time-steps $t-1, t-2$, and so on.

In a DNN, there is no temporal relationship in the way input is processed. Values at time-steps $t, t-1, t-2, \dots $ are all treated distinctly and not as a continuation of the previous time-step values.

$\endgroup$
1
  • $\begingroup$ Doesn't the MLP account for this relationship in the hidden layers? I mean the matrix multiplication to go from the input to the first hidden layer includes all the time steps. $\endgroup$
    – ado sar
    Jan 29 at 0:45
1
$\begingroup$

As noted in LeCun et al.'s Deep Learning paper:

RNNs, once unfolded in time ... can be seen as very deep feedforward networks in which all the layers share the same weights

So, if we ignore how easy they are to train, there is theoretically no real advantage of RNNs over MLPs, on any task, including time series modeling.

Perhaps the key advantage of RNNs is that they share parameters over time. That means they have fewer parameters, and the parameters they do have get used more often. This makes them easier to train (and also allows handling variable-length input sequences, which, as you noted, does not apply when forecasting from a fixed historical window).

Also, the "compression" of the historical information into a state that can pass forward in time, as noted in this excellent answer on quora, may force RNNs to learn general patterns that may generalize better to unseen data.

$\endgroup$
0
$\begingroup$

RNNs have the ability to hold a state, that means the model can learn which information it wants to save and what to delete based on ordering and how you designed the creation and passing of the state (probably worth looking into what an LSTM is), while this would be alot more difficult for a sliding MLP to do (you can think of a sliding MLP as a stateless RNN). Also a sliding MLP would be alot more computation since it needs to recompute an entire context window for each new output, while an RNN could just use the previous state and only do processing on a new singular input or a smaller context window.

Hope that helps!

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .