1
$\begingroup$

I am reading about backpropagation and I wonder why I have to backpropagate.

For example, I would update the network by randomly choosing a weight to change, $w$. I would have $X$ and $y$. Then, I would choose $dw$, a random number from $-0.1$ to $0.1$, for example. Then, I would do two predictions of the neural network and get their losses with the original neural network and one with $w$ changed by $dw$ to get the respective losses $L_{\text{original}}$ and $L_{\text{updated}}$. $L_{\text{updated}} - L_{\text{original}}$ is $dL$. I would update $w$ by $\gamma \frac{d L}{dw}$, where $\gamma$ is the learning rate and $L$ is the loss.

This does not need a gradient backpropagation throughout the system, and must have somehow a disadvantage because no one uses it. What is this disadvantage?

$\endgroup$
  • 1
    $\begingroup$ Algorithms like this exist. Perceptrons, RBMs, Swarm and genetic use this kind of logic (at least it appears similar to me), the problem with this approach is it's not a very good optimisation method and might take long to converge, not to mention the complexity associated with larger networks with lots of connections. BP on the other hand gives straightforward cause and action relationship. $\endgroup$ – DuttaA Jan 18 at 16:50
  • $\begingroup$ Since there's already a good answer I'd like to point out that BP only requires one forward pass (per update) whereas this method requires 2. $\endgroup$ – respectful Jan 18 at 18:23
3
$\begingroup$

The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail.

Using backpropagation to compute the gradient is an exact solution, so why would you use an approximation if the exact computation is equally efficient?

Numeric approximations of the gradient only make sense if exact computation is not possible.

$\endgroup$
  • $\begingroup$ Isn't it much more computationally intensive to compute the gradient every single time in contrast to two forward propagation? $\endgroup$ – Aphrodite Jan 19 at 0:09
  • $\begingroup$ @Aphrodite No, the gradient computed by backpropagation is very efficient, and exact, your approximation is not really faster. $\endgroup$ – Matias Valdenegro Jan 19 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.