3
$\begingroup$

The Turing award is sometimes called Computer Sceince's Nobel Prize. This year's award goes to Bengio, Hinton, and LeCun for their work on artificial neural networks.

The actual work contributed by these authors is, of course, quite technical. It centers around the development of deep neural networks, convolutional neural networks, and effective training techniques. The lay press will tend to simplify these results to the point that they lose meaning.

I would like to have a concise, and yet still precise, explanation of their contributions to share with a lay audience. So, what is a simplified way to explain the contributions of these researchers?

I have my own ideas and will add them if no other satisfactory answer appears. For a "lay" audience, I want to assume someone who had taken a college level course in something scientific but not necessarily computer science. Explanations that are suitable for those with even less background are better still though, as long as they don't lose too much precision.

$\endgroup$
  • $\begingroup$ I have just noticed that you are already linking to an ACM article, which describes their contributions. So, what kind of answer are you actually looking for? $\endgroup$ – nbro Apr 3 at 14:41
  • $\begingroup$ @nbro I've looking for an answer that meets all of these criteria: a): short. Ideally a few sentences. b): does not rely on field specific jargon, to the greatest degree possible. c): Does not lose too much precision (i.e. not just "they made artificial brains."). I think this is quite hard to communicate accurately and briefly to a lay audience. The use case I'm imagining is something like, I bump into a group of first year CS students at the college I teach at, or a biology professor in the lounge, and want to give them an elevator pitch of the recent award. $\endgroup$ – John Doucette Apr 3 at 14:43
  • $\begingroup$ I think it will be difficult to explain their specific contributions without relying on jargon. But, for example, rather than talking about back-propagation, you could talk about "a mathematical technique which allows models to learn from data". I am not sure how you could simplify this further. $\endgroup$ – nbro Apr 3 at 14:47
  • $\begingroup$ @nbro I agree that this is tricky to do. I have some ideas for how this could be simplified further, but because it is tricky, I'm hoping someone has better ideas than mine. As an example, Hinton's backpropagation could be summarized as "We knew for a long time that neural networks could represent complex patterns, but we didn't know how to find those representations just by looking at examples of the pattern. Hinton developed an algorithm to do that." $\endgroup$ – John Doucette Apr 4 at 3:16
  • $\begingroup$ Anyway, I would like to note that Hinton did not really invent back-propagation, which was already known in the 70s. See http://people.idsia.ch/~juergen/who-invented-backpropagation.html. Furthermore, although these researchers have definitely contributed to current situation of AI, there are a lot more people that could have won this award, because of their impact. For example, Sepp Hochreiter and Jürgen Schmidhuber for the development of the LSTM (which is literally everywhere). Maybe they will win it the next years. $\endgroup$ – nbro Apr 4 at 9:06
6
$\begingroup$

The related ACM article describes a few specific technical contributions, which led the ACM to award them.

Geoffrey Hinton

Backpropagation: In a 1986 paper, "Learning Internal Representations by Error Propagation", co-authored with David Rumelhart and Ronald Williams, Hinton demonstrated that the backpropagation algorithm allowed neural nets to discover their own internal representations of data, making it possible to use neural nets to solve problems that had previously been thought to be beyond their reach. The backpropagation algorithm is standard in most neural networks today.

Boltzmann Machines: In 1983, with Terrence Sejnowski, Hinton invented Boltzmann Machines, one of the first neural networks capable of learning internal representations in neurons that were not part of the input or output.

Improvements to convolutional neural networks: In 2012, with his students, Alex Krizhevsky and Ilya Sutskever, Hinton improved convolutional neural networks using rectified linear neurons and dropout regularization. In the prominent ImageNet competition, Hinton and his students almost halved the error rate for object recognition and reshaped the computer vision field.

Yoshua Bengio

Probabilistic models of sequences: In the 1990s, Bengio combined neural networks with probabilistic models of sequences, such as hidden Markov models. These ideas were incorporated into a system used by AT&T/NCR for reading handwritten checks, were considered a pinnacle of neural network research in the 1990s, and modern deep learning speech recognition systems are extending these concepts.

High-dimensional word embeddings and attention: In 2000, Bengio authored the landmark paper, "A Neural Probabilistic Language Model", that introduced high-dimension word embeddings as a representation of word meaning. Bengio's insights had a huge and lasting impact on natural language processing tasks including language translation, question answering, and visual question answering. His group also introduced a form of attention mechanism which led to breakthroughs in machine translation and form a key component of sequential processing with deep learning.

Generative adversarial networks: Since 2010, Bengio's papers on generative deep learning, in particular the Generative Adversarial Networks (GANs) developed with Ian Goodfellow, have spawned a revolution in computer vision and computer graphics. In one fascinating application of this work, computers can actually create original images, reminiscent of the creativity that is considered a hallmark of human intelligence.

Yann LeCun

Convolutional neural networks: In the 1980s, LeCun developed convolutional neural networks, a foundational principle in the field, which, among other advantages, have been essential in making deep learning more efficient. In the late 1980s, while working at the University of Toronto and Bell Labs, LeCun was the first to train a convolutional neural network system on images of handwritten digits. Today, convolutional neural networks are an industry standard in computer vision, as well as in speech recognition, speech synthesis, image synthesis, and natural language processing. They are used in a wide variety of applications, including autonomous driving, medical image analysis, voice-activated assistants, and information filtering.

Improving backpropagation algorithms: LeCun proposed an early version of the backpropagation algorithm (backprop), and gave a clean derivation of it based on variational principles. His work to speed up backpropagation algorithms included describing two simple methods to accelerate learning time.

Broadening the vision of neural networks: LeCun is also credited with developing a broader vision for neural networks as a computational model for a wide range of tasks, introducing in early work a number of concepts now fundamental in AI. For example, in the context of recognizing images, he studied how hierarchical feature representation can be learned in neural networks - a concept that is now routinely used in many recognition tasks. Together with Léon Bottou, he proposed the idea, used in every modern deep learning software, that learning systems can be built as complex networks of modules where backpropagation is performed through automatic differentiation. They also proposed deep learning architectures that can manipulate structured data, such as graphs.

$\endgroup$
  • $\begingroup$ I would like to note that back-propagation was apparently first implemented by Seppo Linnainmaa and not Hinton. $\endgroup$ – nbro Apr 3 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.