0
$\begingroup$

I'd say I have some understanding of backpropagation, however I am not really sure of the total loss being calculated. Let us take the example below: enter image description here

After 1 forward pass when I have to update the parameters, I basically have to find answers to the question - What is the change in loss with respect to the change in a certain parameter i.e. if I were to update parameter w5 then, I need to calculate $$ \frac{\partial L}{\partial w_5} = \frac{\partial L}{\partial o_1} \cdot \frac{\partial o_1}{\partial w_5} $$ So how does the total loss that I calculated after a forward pass even matter? Is it simply for logging(keeping track of)?

Another doubt that I have is: What exactly is this graph indicating? If it is the Loss value for different set of parameter values, then shouldn't it be a discrete function? enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ "So how does the total loss that I calculated after a forward pass even matter?" what do you mean by this? How would you go about computing the partial derivative values without the loss? $\endgroup$
    – Karl
    Commented Mar 21 at 2:26

1 Answer 1

0
$\begingroup$

So how does the total loss that I calculated after a forward pass even matter? Is it simply for logging(keeping track of)?

Decreasing total loss over gradient update steps indicates that the model is learning. For calculating the individual gradients, what matters though are the current values in the nodes. (Since $\frac{\partial L}{\partial L} = 1.$)

Another doubt that I have is: What exactly is this graph indicating? If it is the Loss value for different set of parameter values, then shouldn't it be a discrete function?

No, loss functions are generally continuous with respect to the parameter values. Imagine this as tuning the knob on parameter values, this results a continuous change in the loss. Recall that neural networks are essentially function compositions, and the layers used in practice are all continuous.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .