0
$\begingroup$

Understandably RNNs are very good at solving problems involving audio, video and text processing due to arbitrary input length of this sort of data. What I don't understand is why RNNs are also superior at predicting time series data and why we use them over simple MLP DNNs.

Say I wanted to predict what the value in the time series is at t+1. I would take a window of lets say t-50, t-49...tand then feed loads of sampled training data into a network. I could either choose to have a single LSTM unit remembering the entire window and basing the predictions on that or I could simply make a 50 neuron wide MLP network. What exactly is it about RNNs that makes them better in this scenario or any scenario for that matter?

I understand that the LSTM would have substantially less weights (in this scenario) and should be less computationally intensive, but apart from that I don't see any difference in these two methods.

$\endgroup$
0
$\begingroup$

In an RNN, the output of the previous state is passed as an input to the current state. Intuitively, there is a temporal (time-based) relationship in the way in which input is processed in an RNN. It can understand how the current state was achieved on the basis on the previous values i.e value at time-step t is a result of value at time-steps t-1, t-2 and so on...
In a DNN, there is no temporal relationship in the way input is processed. Values at time-steps t, t-1, t-2... are all treated distinctly and not as a continuation of the previous time-step values.

$\endgroup$
0
$\begingroup$

RNNs have the ability to hold a state, that means the model can learn which information it wants to save and what to delete based on ordering and how you designed the creation and passing of the state (probably worth looking into what an LSTM is), while this would be alot more difficult for a sliding MLP to do (you can think of a sliding MLP as a stateless RNN). Also a sliding MLP would be alot more computation since it needs to recompute an entire context window for each new output, while an RNN could just use the previous state and only do processing on a new singular input or a smaller context window.

Hope that helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.