In an RNN to train it, you need to roll it out, and enter in the history of inputs and the history of expected outcomes.

This doesn't seem like a realistic picture of the brain since this would require, for example, for the brain to store a perfect history of every sense that comes in to it for many time-steps.

So is there an alternative to RNNs that doesn't require this history? Perhaps storing differences or something? Or storing some accumulator?

Perhaps there is a way to calculate with RNNs that doesn't require keeping hold of this history?


You don't necessarily need to roll out the inputs to an RNN; doing so makes it easier to optimize computation (if the sequence length is the same each batch), but it's not a necessity. Furthermore, RNNs (and, incidentally, the brain) doesn't necessarily remember the input history as is; rather, the history is encoded via the RNN's cell state (or states, in the case of LSTMs and other RNN cell architectures with multiple states). Neural Turing Machines (NMTs) and Differential neural computers expand on that concept by also using a larger "memory" storage (in the form of a matrix).

  • $\begingroup$ I don't think that's true. To train an RNN you definitely need to roll it out. To use a pre-trained RNN you don't. $\endgroup$ – zooby Jul 31 '18 at 19:28
  • $\begingroup$ No, you don't need to unroll the input sequence as long as you're building the computation graph as you go (which all modern ML packages will automatically do). Unrolling a fixed-length input sequence lets you (rather, the ML library) optimize the backpropagation, so it's definitely useful if you have the option, but it isn't a necessity $\endgroup$ – velocirabbit Jul 31 '18 at 21:12
  • $\begingroup$ Are you not saying that the RNN is unrolled behind-the-scenes? $\endgroup$ – zooby Jul 31 '18 at 21:27
  • $\begingroup$ Not in the sense that memory space is reserved for it, no. Rather, the graph is built step-by-step and memory is allocated at each step. $\endgroup$ – velocirabbit Jul 31 '18 at 22:10

recurrent-neural-networks act on a sequence of inputs , that does not need to be only a time-sequence for example consider a sequence of characters like a passage or a book.once trained on a sequence of inputs , you could predict the previous and next values of an input-vector at a certain time step.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.