What is the difference between artificial neural networks and deep learning?

Question

I have read many mixed definitions around these two terms. For example, is it right to say deep learning is any ANN with more than two hidden layers?

What are formal definitions for these two?

Answer 1

A few years ago, deep learning was a buzzword, but now is de-facto a standard term or expression, and it's widely used in all research papers, although deep learning is almost never defined rigorously (but this doesn't seem to be a big problem!).

From my experience (this is not just an opinion, of course!), after having read so many papers on the topic, deep learning typically refers to the subset of machine learning algorithms, specifically, gradient descent and back-propagation, applied to neural networks, and, in particular, neural networks with multiple layers. However, there is no consensus on what multiple refers to. In fact, Schmidhuber (co-author of the LSTM), in [1], writes

At which problem depth does Shallow Learning end, and Deep Learning begin? Discussions with DL experts have not yet yielded a conclusive response to this question. Instead of committing myself to a precise answer, let me just define for the purposes of this overview: problems of depth > 10 require Very Deep Learning.

So, although there is typically no agreement on the definition of deep or multiple, there are several problems, such as the vanishing or exploding gradient problems, that arise as the number of layers in a neural network increases, so there are reasons behind the distinction of deep and non-deep/shallow learning.

Note that some research papers (e.g., [1]) distinguish between deep learning applied to neural networks and other types of deep learning, so as not to exclude the layered composition of other models (e.g., SVMs) other than neural networks and the use of machine learning algorithms (e.g. gradient descent) to train them.

To answer your question more directly, neural networks are models or function approximators, while deep learning is the set of machine learning techniques applied to the layered composition of models (typically, neural networks).