I'm trying to learn about neural networks, and I'm interested in gaining a better conceptual understanding of how they work to solve certain problems. I'm having trouble in conceptually understanding how they succeed in doing regression (i.e. predicting continuous variables), however, and wondered if anybody has a good explanation. I know the mathematics of how NNs work, but a clearer conceptual understanding would be helpful.
To give an idea of what I mean by a "conceptual understanding", here's the one that I have for how multilayer NN's with a sigmoid activation function are able to be effective classifiers. Each NN takes the scalar product of its inputs and a set of weights. The weights define a plane in the input space, and the sign of the scalar product indicates which side of the plane the point defined by the inputs is on. The sigmoid activation function outputs 1 if the point is on one side and 0 or -1 (depending on which function is used) if the point is on the other. So the first hidden layer of neurons can be considered to identify which side of each of a group of planes the input point is on. The neurons can also act as AND and OR gates, so subsequent layers of neurons give an output indicating whether the point lies in a region bounded by several of the planes (e.g. a neuron activates only if the point is above one plane and below another, indicating it is in a region of the space associated with one class of points). So if the network learns an appropriate set of planes to bound regions containing different classes of inputs and the AND/OR relations that determine whether a point is in a given region, then it can classify the input points, and this can work for regions with arbitrarily shaped boundaries.
I've not found or been able to think of a similar way of explaining why an NN can perform well in general regression problems (if it's big enough). Does anyone here know a way to explain this?
