Is there any mathematical proof (like in proof of a theorem) based literature out there on neural networks ? Everything is empirically based but no math proof for instance on why certain parameters work ?

By mathematical proof, I mean which parameter works mathematically versus something which does not as in mathematical proof spelled out. This has nothing to do with empirical proof (i.e. something works and here is our guesstimate on why)

  • $\begingroup$ What do you mean? Many books about neural network? What exactly are you looking for? Do you want a proof for neural network? $\endgroup$ – HelloWorld Apr 5 '17 at 0:14
  • $\begingroup$ Ideally yes, but there is not a lot of literature out there. $\endgroup$ – Wizard Programming Apr 5 '17 at 1:14
  • $\begingroup$ ???? ???? Have you searched Google and get some machine learning books? Proofs for neural network are everywhere. Look at "Pattern Recognition and Machine Learning" by Bishop if you want a good one. $\endgroup$ – HelloWorld Apr 5 '17 at 1:15
  • $\begingroup$ I have edited the question and made it more precise. I meant mathematical proof as in proving a theorem. $\endgroup$ – Wizard Programming Apr 5 '17 at 5:11

There is stuff like the Universal Approximation Theorem.

There are also investigations into the loss surface of neural networks.

And classics like this explanation of the vanishing gradient problem.

But I'm afraid the mathematical theory of neural networks only exists in bits and pieces in many different papers. And many of the most important questions can currently only be answered empirically.

|improve this answer|||||
  • 2
    $\begingroup$ Why was this voted down? It's pretty much objectively correct. $\endgroup$ – NietzscheanAI Apr 5 '17 at 10:46

Thomas Cover and David MacKay proofed the capacity of a perceptron. This proof was recently extended to Neural Networks. All of them provide upper bounds for the number of parameters needed to learn something.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.