I understand conceptually how backpropagation works according to the chain rule, and I understand that partial derivatives calculate the rate of change of a function containing multiple variables with respect to one of those variables, the rest being fixed.

What I'm struggling with is what the value from these partial derivatives actually relates to. I found this https://activecalculus.org/multi/S-10-2-First-Order-Partial-Derivatives.html which gives some good examples. But with a NN I'm not sure what units the results of the derivatives relate to.

One of the examples on the website used z = f(x,y) z horizontal distance travelled of a projectile, x initial speed in feet per second, and y was the angle. So if taking the partial derivative with respect to x the results tell us how much the distance travelled changes with respect to the change in speed. So it might be that for every one foot per second increase of the initial speed, we get an increase of 8 feet horizontal travel if using a fixed value for y.

But when calculating the derivatives for backpropagation, does this mean that if we get an answer of (random value) 0.08, this means that for every change of 1 to the non-static variable we would get a change of 0.08 to our output? And what units (if any) do these values relate to?

  • $\begingroup$ that's the right idea, $\frac{\partial f}{\partial \theta_i}$ just represents how much a unit change in $\theta_i$ affects the output f assuming all other variables remain constant and linearity, the chain rule just breaks this up by calculating how $\theta_i$ affects future nodes and then how those nodes affect the function. The units are just $\frac{\text{units of output}}{\text{units of parameter}}$ $\endgroup$
    – quest ions
    Apr 22 at 14:38
  • $\begingroup$ It's similar to the way iteratively reweighted least squares (IRLS) regression works, where a one-step regression is first run, and then the residuals (i.e. error) are regressed iteratively on the input features, and each time, the least squares coeffs represent the delta error, which are added together over iterations to arrive at a solution to the coeffs. $\endgroup$ Apr 22 at 23:03
  • $\begingroup$ @quest-ions That makes sense, and this was kind of the conclusion I came to yesterday after thinking about it for a while. I think I was struggling with not knowing what the "units" were but thinking about them as just units of output/units of parameter makes sense. $\endgroup$
    – Molem7b5
    Apr 23 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.