Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I was designing an Artificial Neural Network a while back, but hit a bump when I got to the backpropagation. I was having trouble making the script choose whether to add or subtract from the weights, when I had a thought.
Does the ANN's training data include the proper output of all the neurons, or just the input and output layers? If it's the latter, could somebody please explain how backpropagation works in a simple way?
Does the ANN's training data include the proper output for every neuron?
The short answer is: no (not usually or directly).
The long answer is that you can train neural networks in different ways. There's supervised, unsupervised, reinforcement learning/training, or even other ways (e.g. online learning).
The most common way of training neural networks is probably in a supervised (and offline) way. In this case, you typically have a dataset of input-output pairs of the form $D = \{(x_1, y_1), \dots, (x_N, y_N) \}$, where we assume that there's an unknown function $f$ such that $f(x_i) = y_i$, so $x_i$ and $y_i$ are the input and corresponding output of $f$. In this case, we want our neural network, which we can denote by $g_{\theta}$ (where $\theta$ are its weights/parameters), to approximate $f$, i.e. $g_\theta \approx f$, for every input and output of $f$. Unfortunately, $D$ does not usually contain all input-output pairs of $f$ (if we had access to all input-output pairs, we wouldn't even need machine learning), so that's why we can only approximate $f$ with our neural network $g_\theta$ (there's also the aspect that $D$ could be noisy, but you can ignore this for now).
So, in this supervised learning context, $y_i$ (also known as the labels or targets) are the supervisory signals to teach the neural network what it's supposed to produce for $x_i$. So, let's say the neural network produces $\hat{y}_i$ rather than $y_i$ when given $x_i$, i.e. $g_\theta(x_i) = \hat{y}_i$. In that case, we have an "error" of $\hat{y}_i - y_i$, but you could also compute the error in other forms other than the difference (depending on the nature of the targets/labels), for example,$|\hat{y}_i - y_i |$ or $(\hat{y}_i - y_i)^2$.
So, once we know the error, we should change the weights of the neural network, i.e. $\theta$, so that the error is smaller the next time we feed $x_i$ as input to $g_\theta$. The way we usually do that is with back-propagation.
You can only understand back-propagation if you understand derivatives and, in particular, how to take derivatives of functions with respect to multiple parameters. Essentially, back-propagation is really just an automatic way of computing derivatives. So, to understand it, you will first need to study a little bit of calculus. This answer is already quite long, so I will not provide more details here, but there are many videos online that explain (even just intuitively) how back-propagation works. So, you should watch them. If you're already familiar with derivatives, maybe this article may be useful (I read it when I was first learning about the topic, but I had already a good knowledge of basic calculus before doing that).
