After watching 3Blue1Brown's tutorial series, and an array of others, I'm attempting to make my own neural network from scratch.

So far, I'm able to calculate the gradient for each of the weights and biases.

Now that I have the gradient, how am I supposed to correct my weight/bias?

Should I:

  1. Add the gradient and the original value?
  2. Multiply the gradient and the original value?
  3. Something else? (Most likely answer)

In addition to this, I've been hearing the term learning rate being tossed around, and how it is used to define the magnitude of the 'step' to descend to minimum cost. I figured this may also play an integral role in reducing the cost.


1 Answer 1


Consider that you have a loss function, and you want to tune your model (network) to decrease the loss. The main concept is to tune parameters in a direction which decreases the loss and gives you a better model. You can imagine a mountain where you should reach to the lower grounds.

There are 2 questions here.

  1. In which direction to move?
  2. How much should we move in that direction?

1. In which direction to move?

By "move" I mean tuning the parameters and therefore changing the model. If you are familiar with the concept of slope or gradient in mathematics, you should move in the direction where the slope is downward the most. The gradient shows the direction with the most slope upward. So, we should move in the opposite of gradient direction. So, we should minus the gradient from the original value, hence the negative sign in the formula below.

2. How much should we move in that direction?

This is defined by the learning rate, which is a number that is multiplied by the gradient. You can imagine that, if the learning rate is big, you are taking bigger footsteps in that direction when coming down the mountain. Similarly, when the learning rate is low, you are taking small footsteps coming down the mountain.

So, consider you have $\nabla_w$ (the gradient of the parameters $w$, with respect to the loss function $L$). Let the learning rate be $\gamma$ (for example, $\gamma= 0.1$). The formula to update the parameters would be

$$ w_{\text{new}} = w_{\text{old}} - \gamma * \nabla_{w_{\text{old}}} $$

Note that gradient descent is not the only way to optimize your model. Many other gradient-based approaches exist, for example, stochastic gradient descent, Adam, RMSprop, conjugate gradients, etc. There other methods, like evolutionary methods (genetic algorithm, for example), that use different concepts.

Also, note that the learning rate does not necessarily have to be fixed, so it can be tuned during the training, if you want or need.


You must log in to answer this question.