2
$\begingroup$

I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the backprop algorithm know which weights to change or which paths to take? The way I did it, it always took all of the paths/weights and changed them all. So how does the algorithm know which paths to take, which weights to change, and whether to add or subtract X amount from said weight?

$\endgroup$
1
  • 4
    $\begingroup$ Basically using chain rule. Check this video and if it is your taste, the following one on the playlist with the mathematical details to get a better understanding of the process. This one might also help. $\endgroup$
    – serali
    Oct 6, 2021 at 12:40

1 Answer 1

3
$\begingroup$

@serali is correct, there are many resources to describe this process. I'm going to talk about it in a very general way.

Disclaimer: this is a work in progress, it is not yet complete.

Question:
How do we know what weights to change in the network?

General Answer:
This is a very good question. This is a key component of ML, and even small improvements there would make a huge difference for the overall community. There are gigawatt-hours of electricity spent using the current answers to this question.

Training means you have an input that you want the network to transform into an output, and it means you have an output to compare it against.

The difference between what it makes, and what it should make, the error, can be run in reverse through the network. As you go through that reverse run, you can get an error associated with each weight. if you move that error to zero, by changing the weight, then the network can be less bad.

Some variations:
Here are some approaches for relating the per-weight errors to the new weights.

Approaches:

  • (sgd link) take just a few or one sample at random and compute the error, and then move weights in that direction, usually only a small percentage of it. The intention is that each new update occurs closer to the optimum, and works well in lower-noise cases.
  • (conjugate gradient link, ) find a line along which to minimize, find the minimum, explore around that minimum to find a new direction for minimization. For quadratic problems (shaped like parabola) it gives good solutions quickly.
  • (momentum link) take a running average of the last several weight changes, and do most of the movement in that direction
  • (regularization link) penalize weights that are too high, push back against learning them, tries to keep weights in smaller subset of the space unless pushed there by the data.
  • (dropout link) randomly set internal responses of neurons to zero, to force a robust and distributed representation of the processing in the network. Keeps on network from "dominating the show" because in practice it tends to do more harm than good for generalization.
  • (rotated locking link) aka stochastic depth, randomly freeze layers in the architecture to avoid most learning occurring only in the last few layers, aka diminishing gradients.
  • (pca in vectorized weights) given a batch of errors for weights, perform pca and only retain highest x percent of eigenvalues, move in that direction.
  • (entropy-scaled momentum in pca elements) given batch of errors for weights, perform svd, and then map weight changes per sample to the largest eigenvectors associated with it
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .