How is a neural network having the "deep" adjective actually distinguished from other similar networks?


The difference is mostly in the number of layers.

For a long time, it was believed that "1-2 hidden layers are enough for most tasks" and it was impractical to use more than that, because training neural networks can be very computationally demanding.

Nowadays, computers are capable of much more, so people have started to use networks with more layers and found that they work very well for some tasks.

The word "deep" is there simply to distinguish these networks from the traditional, "more shallow" ones.

  • $\begingroup$ If I understood "deep learning" correctly, there are some parameters (weights) that are tied together, thus reducing the parameter space. Normal NNs can't do that. $\endgroup$ – Raphael Aug 14 '17 at 5:55
  • $\begingroup$ @Raphael, re: some parameters (weights) that are tied together, thus reducing the parameter space, do you mean Convolutional Neural Networks? $\endgroup$ – publicgk Jan 15 '18 at 11:18
  • $\begingroup$ @publicgk That's what I saw, yes. $\endgroup$ – Raphael Jan 15 '18 at 13:02
  • $\begingroup$ I have always heard "deep learning" is when you use very large datasets. Is this a mistake and the size of the data doesn't matter - or is do people associated deep learning with huge data sets because that is what is required for training. $\endgroup$ – Steven Sagona Feb 20 '19 at 0:14
  • $\begingroup$ I would like to point out that this "belief" that a neural network with just 1 hidden layer can compute any function is actually been proven (see e.g. neuralnetworksanddeeplearning.com/chap4.html). I think you should have explained a little bit more why more than 1 hidden layer is then "convenient". $\endgroup$ – nbro Feb 23 '19 at 22:14

A deep neural network is just a (feed-forward) neural network with many layers.

However, deep belief networks, Deep Boltzman networks, etc., are not considered (debatable) deep neural networks, as their topology is different (they ave undirected networks in their topology).

See also this: https://stats.stackexchange.com/a/59854/84191.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.