I am new to neural-network and I am trying to understand mathematically what makes neural networks so good at classification problems.
By taking the example of a small neural network (for example, one with 2 inputs, 2 nodes in a hidden layer and 2 nodes for the output), all you have is a complex function at the output which is mostly sigmoid over a linear combination of the sigmoid.
So, how does that make them good at prediction? Does the final function lead to some sort of curve fitting?
Neural networks are good at classifying. In some situations that comes down to prediction, but not necessarily.
The mathematical reason for the neural networks prowess at classifying is the universal approximation theorem. Which states that a neural network can approximate any continuous real-valued function on a compact subset. The quality of the approximation depends on the number of neurons. It has also been shown that adding the neurons in additional layers instead of adding them to existing layers improves the quality of the approximation faster.
Add to that the not well-understood effectiveness of the backpropagation algorithm and you have a setup then can actually learn the function that the UAT promises or something close.
With Neural Networks you simply classify datas. If you classify correctly, so you can do future classifications.
How It Works?
Simple neural networks like Perceptron can draw one decision boundary in order to classify datas.
For example suppose you want to solve simple AND problem with simple Neural Network. You have 4 sample data containing x1 and x2 and weight vector containing w1 and w2. Suppose initial weight vector is [0 0]. If you made calculation which depend on NN algoritm. At the end, you should have a weight vector [1 1] or something like this.
Please focus on the graphic.
It says: I can classify input values into two classes (0 and 1). Ok. Then how can I do this? It is too simple. First sum input values (x1 and x2).
0+0=0
0+1=1
1+0=1
1+1=2
It says:
if sum<1.5 then its class is 0
if sum>1.5 then its class is 1
Neural networks excel at a variety of tasks, but to get an understanding of exactly why, it may be easier to take a particular task like classification and dive deeper.
In simple terms, machine learning techniques learn a function to predict which class a particular input belongs to, depending on past examples. What sets neural nets apart is their ability to construct these functions that can explain even complex patterns in the data. The heart of a neural network is an activation function like Relu, which allows it to draw some basic classification boundaries like: 
By composing hundreds of such Relus together, neural networks can create arbitrarily complex classification boundaries, for example:
In this article, I try to explain the intuition behind what makes neural networks work: https://medium.com/machine-intelligence-report/how-do-neural-networks-work-57d1ab5337ce
In Neural Networks we consider everything in high dimension and try to find a hyperplane that classify them by small changes...
Probably it is hard to prove that it works but intuition says if it can be classified you can do it by add a relaxed plane and let it move amongst data to find a local optimum...
