Actually, the hierarchical learning explanation given by mindcrime is not that acceptable anymore (This was also indicated by Ian Goodfellow). Since there are neural networks with 150 layers or more, and this explanation does not make sense for such neural networks. However, we can think of it as solving the knots of high dimensional manifolds, i.e. we transform the input into high dimensional space, and this helps us to find a better representation of the data.
A geometric interpretation was explained as such in the book Deep Learning with Python by François Chollet:
...you can interpret a neural network as a very complex geometric transformation in a high-dimensional space, implemented via a long series of simple steps...
Imagine two sheets of colored paper: one red and one blue. Put one on top of the other. Now crumple them together into a small ball. That crumpled paper ball is your input data, and each sheet of paper is a class of data in a classification problem. What a neural network (or any other machine-learning model) is meant to do is figure out a transformation of the paper ball that would uncrumple it, so as to make the two classes cleanly separable again. With deep learning, this would be implemented as a series of simple transformations of the 3D space, such as those you could apply on the paper ball with your fingers, one movement at a time. Uncrumpling paper balls is what machine learning is about: finding neat representations for complex, highly folded data manifolds. At this point, you should have a pretty good intuition as to why deep learning excels at this: it takes the approach of incrementally decomposing a complicated geometric transformation into a long chain of elementary ones, which is pretty much the strategy a human would follow to uncrumple a paper ball. Each layer in a deep network applies a transformation that disentangles the data a little—and a deep stack of layers makes tractable an extremely complicated disentanglement process.
I suggest you to read this brilliant blog post to learn about the topological interpretation of deep learning.
Also, this toy interactive code may help you.
In the context of machine learning, the concept of a manifold can be illustrated as in the following figure.
In the first part, data are 3-dimensional. However, we can find a transformation to get the second image, which shows that data is actually artificially high dimensional, i.e. it is a 2-dimensional manifold in 3-D space. This example may be thought of as a classification problem, and colors may represent classes, and we can find a trivial representation of the data for classification.
Another example could be following figures from the blog I mentioned. In here, this classification problem cannot be solved without having a layer that has 3 or more hidden units, regardless of depth. So the notion of high dimensional transformation is important.
We can map this data to 3-D, and find a plane to separate them.
