I was able to find the original paper on LSTM, I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?


It was difficult to find because recurrent network designs predate LSTM extensions of that earlier idea by decades.

Although the term recurrent was not yet used as a primary description of the technology advancement, recurrence was an essential feature of the theoretical treatment of artificial networks that learned actions in Attractor dynamics and parallelism in a connectionist sequential machine, by Michael I. Jordan, 1986, Cogitive Science Conference, pp 531-546. This seminal article has been reprinted in the book Artificial neural networks, pp 112-127, IEEE Press Piscataway, 1990, ISBN 0-8186-2015-3.

Three years later, Barak A. Pearlmutter of the National University of Ireland dually published Learning state space trajectories in recurrent neural networks in Proceedings of IJCNN, vol.2, pp 365-372, 1989 and Neural Computation, vol. 1, pp 263-269, 1989.

That same year, 1989, Finite-state automata and simple recurrent networks by Cleeremans, A., Servan-Schreiber, D., and McClelland, J. L. was also published in Neural Computation, pp , 1:372-381.

I'm guessing that there was a call for papers on the topic from the editorial staff of Neural Computation, and the term may have originated from those editors or something written earlier that decade, but the word did not appear in any of the titles in the bibliographies of these publications.

LSTMs were not invented for another eight years to solve the problem with retention and the predictive distinction inherent in the staleness of learned information. Attention based memory is a level of sophistication further, and current attempts to determine the applicability of learned information on the basis of semantic matching is underway in the fields of conversant computers and in adaptive walking, driving, and piloting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.