2
$\begingroup$

I am quite new in the AI field. I am trying to create a neural network, in a language (Dart) where I couldn't find examples or premade libraries or tutorials. I've tried looking online for a strictly "vanilla" python implementation (without third-party libraries), but I couldn't find any.

I've found a single layer implementation, but it's done only with matrices and it's quite cryptic for a beginner.

I've understood the idea between the feed forwarding, a neuron calculates the sum of its inputs, adds a bias and activates it.

But I couldn't find anything a neuron-level explanation of the math behind backpropagation. (By neuron-level I think of the math down to the single neuron as a sequence of operations instead of multiple neurons treated as matrices).

What is the math behind it? Are there any resources to learn it that are suitable as a beginner?

Improve this question
$\endgroup$
5
  • $\begingroup$ backpropagation is a bit harder. You basically calculate a loss function between the values you predict and the labels, and then propagate it with some form of gradient descend to the weights in different layers. Also implementing it without matrices might be very inefficient. If you search online for "backpropagation simple implementation" or "backpropagation math step by step" or something else I'm sure you will find plenty of examples. $\endgroup$
    – Miguel Saraiva
    Jan 6, 2020 at 21:46
  • $\begingroup$ You can think of the matrix math as just a notation for the several individual multiplications, etc. not magic. $\endgroup$
    – George White
    Jan 6, 2020 at 22:11
  • $\begingroup$ Here is an example with values which may help to explain what is happening. mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example $\endgroup$
    – lfgtm
    Jan 6, 2020 at 22:59
  • $\begingroup$ Check Michael Nielsen's book Neural Networks and Deep Learning. Math of backpropagation is discussed at length and he does not use any libraries - just Python. $\endgroup$
    – serali
    Jan 7, 2020 at 12:49
  • $\begingroup$ you begin by getting the loss of the output then you propagate and apply gradients back through the layers using the chain rule, to do a few layers of this statically isnt hard just verbose, to do so for arbitrary numbers of layers means your looking at implementing something close to autodiff $\endgroup$
    – nickw
    Jan 8, 2020 at 15:01

1 Answer 1

Reset to default
1
$\begingroup$

Backpropagation is actually a lot easier than it is made out to be - if you have a basic understanding of calculus and the chain rule, and the single multi-variable calculus rule that to combine 2 gradient vectors, you simply add them.

This is hands down the best walk through of back prop I've found on the internet. If you are still confused after that, feel free to ask me any further questions. Here is also a quick forward and backward pass example I made for a simple CNN (only a few layers though, and the gradient only goes back to channel 1 of filter 1)backprop

Improve this answer
$\endgroup$
1
  • $\begingroup$ The Stanford university video you liked (ep 4 and 6) are indeed the clearest and simplest explanation on the topic I've found online! Thanks again! $\endgroup$
    – Fabrizio
    Jan 9, 2020 at 20:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .