2
$\begingroup$

We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the gradient:

  1. BackPropagation
  2. Calculating gradient of loss function by calculus

I found many resources for understanding backpropagation. The 2nd method I am referring to is the image below(taken for a specific example, e is the error: difference between target and prediction): enter image description here

Also, the proof was mentioned in this paper:here

Moreover, I found this method while reading this blog.(You might have to scroll down to see the code: gradient = X.T.dot(error) / X.shape[0] )

My question is are the two methods of finding gradient of cost function same? It appears different and if yes, which one is more efficient( though one can guess it is backpropagation)

Would be grateful for any help. Thanks for being patient(it's my 1st time learning ML).

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ BackPropagation is called so because it uses the special chain rule of calculus to back propagate gradients and so yes both are same. Also, if other method exist to back propagate in a similar way it will also be called back propagation. $\endgroup$
    – user9947
    Mar 31, 2020 at 12:13
  • $\begingroup$ But sir, if both are same, why do we use backPropagation, isn't it very lengthy $\endgroup$
    – Hrushi
    Mar 31, 2020 at 13:41
  • 1
    $\begingroup$ Backpropagation is a method/process of propagating the loss back to previous layers, whereas gradient descent (and it's variants like SGD,RMSprop etc) is a method to optimize the neural network to reduce the loss. Loss is less when we "descent" to a valley point, which we calculate through gradients or slope (ie. derivatives) and we send this information back using backprop where different channels receive different value of the gradient depending on their contribution. We can also send info other than gradients using backprop, there are also methods which don't use one or other, or either $\endgroup$
    – SajanGohil
    May 4, 2020 at 18:09

1 Answer 1

Reset to default
1
$\begingroup$

I'm pretty sure they're the same thing. Both backpropagation and gradient descent's essential ideas are to calculate the partial derivative of the cost with respect to each weight, and subtract the partial derivative times the learning rate.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes, I believe both are the same. They are different methods and backpropagation is a faster algorithm. This is mentioned and explained why in [here] (neuralnetworksanddeeplearning.com/…) $\endgroup$
    – Hrushi
    Apr 1, 2020 at 3:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .