I was able to find the original paper on LSTM, I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?


The two tech reports below both call RNNs explicitly "recurrent net(work)s". One of them predates the paper mentioned in the accepted answer.

  1. Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations by error propagation. Tech. rep. ICS 8504. San Diego, California: Institute for Cognitive Science, University of California.
  2. Jordan, Michael I. (May 1986). Serial order: a parallel distributed processing approach. Tech. rep. ICS 8604. San Diego, California: Institute for Cognitive Science, Universityof California.

Jordan was a student of Rumelhart, so I would lean on identifying [1] as the paper introducing RNNs. With the caveat that the first sentence in the section "Recurrent Nets" reads:

We have thus far restricted ourselves to feedforward nets. This may seem like a substantial restriction, but as Minsky and Papert point out, there is, for every recurrent network, a feedforward network with identical behavior (over a finite period of time).

This is interesting for two reasons:

  1. After this sentence, he then goes on to show how RNNs can be unrolled and the error propagated back. Not a full-fledged BPTT yet, though.
  2. The sentence shows that the idea of recurrence (and unrolling) has been around since at least 1969.

Unfortunately I don't have access to Minsky and Papert (1969), so I cannot follow this line any further.

| improve this answer | |

It was difficult to find because recurrent network designs predate LSTM extensions of that earlier idea by decades.

Although the term recurrent was not yet used as a primary description of the technology advancement, recurrence was an essential feature of the theoretical treatment of artificial networks that learned actions in Attractor dynamics and parallelism in a connectionist sequential machine, by Michael I. Jordan, 1986, Cogitive Science Conference, pp 531-546. This seminal article has been reprinted in the book Artificial neural networks, pp 112-127, IEEE Press Piscataway, 1990, ISBN 0-8186-2015-3.

Three years later, Barak A. Pearlmutter of the National University of Ireland dually published Learning state space trajectories in recurrent neural networks in Proceedings of IJCNN, vol.2, pp 365-372, 1989 and Neural Computation, vol. 1, pp 263-269, 1989.

That same year, 1989, Finite-state automata and simple recurrent networks by Cleeremans, A., Servan-Schreiber, D., and McClelland, J. L. was also published in Neural Computation, pp , 1:372-381.

I'm guessing that there was a call for papers on the topic from the editorial staff of Neural Computation, and the term may have originated from those editors or something written earlier that decade, but the word did not appear in any of the titles in the bibliographies of these publications.

LSTMs were not invented for another eight years to solve the problem with retention and the predictive distinction inherent in the staleness of learned information. Attention based memory is a level of sophistication further, and current attempts to determine the applicability of learned information on the basis of semantic matching is underway in the fields of conversant computers and in adaptive walking, driving, and piloting.

| improve this answer | |
  • 1
    $\begingroup$ This answer cannot be right. The paper "Learning internal representations by error propagation" (1985) already mentions the expression "recurrent neural networks", which have been discussed in 1969 by Minsky and Papert. The question isn't about which paper introduced the expression "RNN", which isn't any of the papers you mention anyway. $\endgroup$ – nbro Mar 10 at 13:34

Hopfield networks, a special case of RNNs, were first proposed in 1982: https://www.pnas.org/content/79/8/2554

Otherwise (shameless plug, I am the author) a non-technical timeline for NLP can be found here: https://blog.exxcellent.de/ki-machine-learning

enter image description here

| improve this answer | |
  • $\begingroup$ You say "a non-technical timeline for NLP can be found here:". Are you talking about a timeline for when RNNs were applied to NLP tasks? $\endgroup$ – nbro Nov 4 at 11:49
  • 1
    $\begingroup$ No, the timeline shows the dates of the papers describing them. TR-808 was the introduction of the famous drum computer, referred to in the German text. 1982-86 were the papers on Hopfield networks and RNNs. 1995-97 the papers on LSTMs. And 1999 is the date the first GPU was launched. If you have corrections or comments, I would love to hear them. $\endgroup$ – AlDante Nov 5 at 14:25

Warren McCulloch and Walter Pitts talk about recurrent neural nets in their paper McCulloch, W.S., Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 115–133 (1943). https://doi.org/10.1007/BF02478259.

They finish their introduction with the paragraph:

The nervous system contains many circular paths, whose activity so regenerates the excitation of any participant neuron that reference to time past becomes indefinite, although it still implies that afferent activity has realized one of a certain class of configurations over time. Precise specification of these implications by means of recursive functions, and determination of those that can be embodied in the activity of nervous nets, completes the theory.

Their paper contains a section titled:

  1. The Theory: Nets Without Circles.

in which they introduce feed-forward (nets without cycles) and recurrent (nets with cycles) networks, and the next section, titled

  1. The Theory: Nets with Circles.

in which they prove a few theorems about recurrent neural networks.

Marvin Minsky quotes them, and discusses recurrent neural networks extensively throughout his book, Computation: Finite and Infinite Machines (1967). Prentice Hall, ISBN: 0131655639,9780131655638

I am not sure, are there earlier references.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.