3
$\begingroup$

In an RNN to train it, you need to roll it out, and enter in the history of inputs and the history of expected outcomes.

This doesn't seem like a realistic picture of the brain since this would require, for example, for the brain to store a perfect history of every sense that comes in to it for many time-steps.

So is there an alternative to RNNs that doesn't require this history? Perhaps storing differences or something? Or storing some accumulator?

Perhaps there is a way to calculate with RNNs that doesn't require keeping hold of this history?

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
1
$\begingroup$

You don't necessarily need to roll out the inputs to an RNN; doing so makes it easier to optimize computation (if the sequence length is the same each batch), but it's not a necessity. Furthermore, RNNs (and, incidentally, the brain) doesn't necessarily remember the input history as is; rather, the history is encoded via the RNN's cell state (or states, in the case of LSTMs and other RNN cell architectures with multiple states). Neural Turing Machines (NMTs) and Differential neural computers expand on that concept by also using a larger "memory" storage (in the form of a matrix).

Improve this answer
$\endgroup$
4
  • $\begingroup$ I don't think that's true. To train an RNN you definitely need to roll it out. To use a pre-trained RNN you don't. $\endgroup$
    – zooby
    Jul 31, 2018 at 19:28
  • $\begingroup$ No, you don't need to unroll the input sequence as long as you're building the computation graph as you go (which all modern ML packages will automatically do). Unrolling a fixed-length input sequence lets you (rather, the ML library) optimize the backpropagation, so it's definitely useful if you have the option, but it isn't a necessity $\endgroup$
    – velocirabbit
    Jul 31, 2018 at 21:12
  • $\begingroup$ Are you not saying that the RNN is unrolled behind-the-scenes? $\endgroup$
    – zooby
    Jul 31, 2018 at 21:27
  • $\begingroup$ Not in the sense that memory space is reserved for it, no. Rather, the graph is built step-by-step and memory is allocated at each step. $\endgroup$
    – velocirabbit
    Jul 31, 2018 at 22:10
0
$\begingroup$

recurrent-neural-networks act on a sequence of inputs , that does not need to be only a time-sequence for example consider a sequence of characters like a passage or a book.once trained on a sequence of inputs , you could predict the previous and next values of an input-vector at a certain time step.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .