1
$\begingroup$

I am new to deep learning. I have doubts on modifying bias values during back propagation. My doubts are

  1. Does the back propagation algorithm modifies the weigh values and bias values in the same pass?
  2. How does the algorithm decide whether it has to change the weight value or bias value to reduce the error in a pass?
  3. Will the learning rate same for bias and weights?

Thanks!

$\endgroup$
2
$\begingroup$

Does the back propagation algorithm modifies the weigh values and bias values in the same pass?

Yes.

How does the algorithm decide whether it has to change the weight value or bias value to reduce the error in a pass?

It differentiates the loss function (like MSE) with respect to the weights and biases, that is, it finds the partial derivative of the loss function with respect to each of the parameters. You can think of the partial derivative of the loss function with respect to one of the parameters of the model as representing the "contribution" of that parameter to the loss of the model.

The partial derivatives of the loss function with respect to each of the parameters (weights and biases) is collectively called the gradient, which is thus a vector of $N$ partial derivatives, where $N$ is the number of parameters of the model.

Will the learning rate same for bias and weights?

It is usually the same, but, in theory, nobody prevents you from updating the biases using a different learning rate, so you could use a different learning rate to update the biases and weights.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.