2
$\begingroup$

In recent years we have seen quite a lot of impressive display of Deep Neural Network (DNN), as demonstrated most famously by AlphaGo and its cousin programs.

But if I understand correctly, deep neural network is just a normal neural network with a lot of layers. We know about the principles of the neural network since the 1970s (?) and deep neural network is just the generalization of a one-layer neural network to many.

From here it doesn't seem like the recent explosion of DNN has anything to do with a theoretical breakthrough. It seems like DNN successes can be entirely attributed to better hardware and more data, and not to any new theoretical insights or better algorithms.

I would go even as far as saying that there are no new theoretical insights/algorithms that contribute significantly to the DNN recent successes; that all theoretical underpinning of DNN was done in the 1970s or prior.

An I right on this?

| improve this question | |
$\endgroup$
  • 2
    $\begingroup$ Related: datascience.stackexchange.com/questions/14352/… $\endgroup$ – Neil Slater Nov 13 '19 at 7:30
  • $\begingroup$ @NeilSlater, not quite the same. As I am asking about how much weight theoretical advancement has ( or is any) in contributing to DNN . $\endgroup$ – Graviton Nov 13 '19 at 9:50
  • $\begingroup$ What are you willing to count as "theoretical advancement"? It is possible to claim everything is just book-keeping since the invention of calculus and the chain rule (in 17th century). Backpropagation did not invent this theory, just applied it. Optimisation through gradient descent is also a very old theoretically. Neural networks seen in that light are primarily an invention of applied maths/engineering, not theoretical constructs . . . so I would like to understand your threshold for what you consider as a theory? $\endgroup$ – Neil Slater Nov 13 '19 at 10:27
  • $\begingroup$ @Graviton, Several items in the link above can be considered as theoretical advancements. Using ReLU instead of smooth functions, or the theory section above discussing local vs global minima in higher dimensional functions are theoretical; and the introduction of dropout layer is also in the list. But are they revolutionary? That is a different question. $\endgroup$ – serali Nov 13 '19 at 10:30
  • $\begingroup$ @serali I agree with you that the question whether those advancements are revolutionary is different from the linked SE question, and hence this is my question here $\endgroup$ – Graviton Nov 13 '19 at 11:21
1
$\begingroup$

The first neural network machine was the stochastic neural analog reinforcement calculator (SNARC), built in the 1950s. As you can see, it's pretty old. After that, there were several advances regarding backpropagation and the vanishing gradient problem. However, the ideas itself are not novel. Simply put, we have the data and processing power today that we did not have back then.

You could look at the Wikipedia timeline.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.