5
$\begingroup$

I'm trying to learn about neural networks, and I'm interested in gaining a better conceptual understanding of how they work to solve certain problems. I'm having trouble in conceptually understanding how they succeed in doing regression (i.e. predicting continuous variables), however, and wondered if anybody has a good explanation. I know the mathematics of how NNs work, but a clearer conceptual understanding would be helpful.

To give an idea of what I mean by a "conceptual understanding", here's the one that I have for how multilayer NN's with a sigmoid activation function are able to be effective classifiers. Each NN takes the scalar product of its inputs and a set of weights. The weights define a plane in the input space, and the sign of the scalar product indicates which side of the plane the point defined by the inputs is on. The sigmoid activation function outputs 1 if the point is on one side and 0 or -1 (depending on which function is used) if the point is on the other. So the first hidden layer of neurons can be considered to identify which side of each of a group of planes the input point is on. The neurons can also act as AND and OR gates, so subsequent layers of neurons give an output indicating whether the point lies in a region bounded by several of the planes (e.g. a neuron activates only if the point is above one plane and below another, indicating it is in a region of the space associated with one class of points). So if the network learns an appropriate set of planes to bound regions containing different classes of inputs and the AND/OR relations that determine whether a point is in a given region, then it can classify the input points, and this can work for regions with arbitrarily shaped boundaries.

I've not found or been able to think of a similar way of explaining why an NN can perform well in general regression problems (if it's big enough). Does anyone here know a way to explain this?

$\endgroup$
  • $\begingroup$ NN is a non-linear model. Think of it like a non-linear regression, then you won't have a problem. $\endgroup$ – SmallChess May 9 '17 at 14:21
  • $\begingroup$ Would you be able to elaborate a bit more? $\endgroup$ – PeterW May 9 '17 at 14:34
  • $\begingroup$ @SmallChess Answer the OP $\endgroup$ – quintumnia May 11 '17 at 10:22
5
$\begingroup$

I came across this article which explains how a neural network can approximate any function, which pretty much answers my question.

I suppose connections between neurons do not generally arrange themselves in the way described, though, so it would still be interesting to get insight into whether there are patterns of how the inter-neuron connections arrange themselves in neural networks trained to solve actual problems.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.