This is a cross-post, as I didn't get any answers on Stats SE and I am hoping that it gets more attention here.

At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight Normalization, the equation for the gradient of the output with respect to the weights is given as:

$$ \triangledown y_{BN} (X; w, \gamma, \beta) = \frac{X}{\sigma(X)}\gamma g'(z). $$

Can someone break down into smaller steps on how the author got to that equation?

  • $\begingroup$ Hi and welcome to AI SE! Meanwhile, if you don't receive an answer, what you can do is try to understand the formula step by step. To do that, you first need to understand all the terms in it. For example, what is $X$? It should be the input. $g$ should be the non-linearity? What are all other terms? $\endgroup$ – nbro Feb 27 '20 at 15:50
  • $\begingroup$ Watch this: youtu.be/d14TUNcbn1k?t=354. If you are still unsure of anything in relation to backprop, respond here and I'd be happy to help $\endgroup$ – Recessive Feb 28 '20 at 5:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.