I have read many mixed definitions around these two terms. For example, is it right to say deep learning is any ANN with more than two hidden layers?
What are formal definitions for these two?
Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.Sign up to join this community
A few years ago, deep learning was a buzz word, but now is de-facto a standard term or expression, and it's widely used in all research papers, although deep learning is almost never defined rigorously (but this doesn't seem to be a big problem!).
From my experience (this is not just an opinion, of course!), after having read so many papers on the topic, deep learning typically refers to the subset of machine learning techniques (specifically, gradient descent and back-propagation) applied to neural networks, and, in particular, neural networks with multiple layers. However, there is no consensus on what multiple refers to. In fact, Schmidhuber (co-author of the LSTM), in , writes
At which problem depth does Shallow Learning end, and Deep Learning begin? Discussions with DL experts have not yet yielded a conclusive response to this question. Instead of committing myself to a precise answer, let me just define for the purposes of this overview: problems of depth > 10 require Very Deep Learning.
In other words, although there is typically no agreement on the definition of deep or multiple, there are several problems (such as the vanishing or exploding gradient problems) that arise as the number of layers in a neural network increases, so there are reasons behind the distinction of deep and non-deep learning.
Note that some research papers (e.g. ) distinguish between deep learning applied to neural networks and other types of deep learning, so that not to exclude the layered composition of other models (e.g. SVMs) other than neural networks and the use of machine learning algorithms (e.g. gradient descent) to train them.
To answer your question more directly, neural networks are models or function approximators, while deep learning is the set of machine learning techniques applied to the layered composition of models (typically, neural networks).