I am trying to dissect the paper Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks.

Unfortunately, because my math is a little bit rusty, I got a little bit stuck with the proof. Could you provide me with some clarification about proof of the topic?

What I understand is that we introduce, instead of the weight vector $w$ a scalar $g$ (magnitude of original $w$?) and $\frac{v}{\|v\|}$ (direction of original $w$?) of the vector.

$$\nabla_{g} L=\frac{\nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\|\mathbf{v}\|}$$


$$\nabla_{\mathbf{v}} L=\frac{g}{\|\mathbf{v}\|} \nabla_{\mathbf{w}} L-\frac{g \nabla_{g} L}{\|\mathbf{v}\|^{2}} \mathbf{v}$$

What I am not really sure about is:

If the gradients are noisy (does this mean that in some dimension we have small and in some high curvature or that error noise differs for very similar values of $w$?) the value will quickly increase here and effectively limit the speed of descent by decreasing value of ($\frac{g}{\|v\|}$). This means that we can choose larger learning rates and it will somehow adjust the effect of the learning rate during the training.

And what I completely miss is:

$$\nabla_{\mathbf{v}} L=\frac{g}{\|\mathbf{v}\|} M_{\mathbf{w}} \nabla_{\mathbf{w}} L$$


$$\mathrm{M}_{\mathrm{w}}=\mathrm{I}-\frac{\mathrm{w} \mathrm{w}^{\prime}}{\|\mathrm{w}\|^{2}}$$

It should somehow explain the reasoning behind the idea of the final effects. Unfortunately, I don't really understand this part of the paper, I probably lack some knowledge about Linear Algebra.

Can you verify that my understanding of the paper is correct?

Can you recommend some sources (books/videos) to help me to understand the second part of proof (related to the second set of formulas?)


1 Answer 1


If the gradients are noisy (does this mean that in some dimension we have small and in some high curvature or that error noise differs for very similar values of w?)

Gradients being noisy means that they are "inconsistent" across different epochs / training steps. With that I mean that they'll sometimes point in one direction, later in a different (maybe the opposite) direction, etc., that the gradients at different time steps give inconsistent/conflicting information, that we don't consistently keep following the same direction through gradient descent but keep jumping all over the place. Note that this is across epochs, it's not across different dimensions within the same epoch.

So, for example, we might have a vector of gradients that looks like $[1, 1, 1, \dots, 1]$ at one time step (i.e., positive gradients in all dimensions), and the next training step get a gradient more like $[-1, -1, -1, \dots, -1]$ (this example is pretty much the "most extreme" kind of noise you can have, completely conflicting directions, in reality it'd generally be less extreme).

Such "noisy gradients" can exist because, in practice, we almost always use estimates of the gradient of our loss function, rather than the true gradients (which we'll generally not even be able to compute). For example, if we have a very large training dataset in a supervised learning setting, we'd ideally use the gradient of the loss function computed across the complete dataset. That tends to be computationally expensive (requires forwards and backwards passes for every single instance in the dataset), so in practice we'll often only use a small minibatch to estimate the gradient; this can result in widely different estimates of the gradient for different minibatches. (note: there can also be different reasons for using minibatches rather than full dataset, such as avoiding overfitting, but that's not too important to consider for this specific question)

A different example of a setting where we can have noisy gradients is in Deep Reinforcement Learning. There, computing the loss function itself is often rather noisy (in the sense that our own predictions, which tend to still be incorrect during the training phase, are a component of the targets that we're updating towards), so our estimates of the gradients will also be noisy.

Derivation for your second question, about the gradient of the loss with respect to $\mathbf{v}$:

We start with the following, from Equation (3) in the paper (this gradient you understand, right?):

$$\nabla_{\mathbf{v}} L = \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \nabla_g L}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v}$$

Note that Equation (3) also gives:

$$\nabla_{g} L = \frac{\nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}$$

If we plug that into the previous Equation we get:

\begin{aligned} \nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \frac{\nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v} \\ &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert^3} \mathbf{v} \\ \end{aligned}

The $\frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L$ term that shows up before the minus can also be isolated in the term after the minus:

\begin{aligned} \nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v} \\ &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert} \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert} \\ \end{aligned}

Equation (2) tells us that $\mathbf{w} = \frac{g}{\vert \vert \mathbf{v} \vert \vert} \mathbf{v}$, which means that $\frac{\mathbf{w}}{g} = \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}$. Plugging this into the above leads to:

\begin{aligned} \nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w}}{g} \cdot \frac{\mathbf{w}}{g} \\ \end{aligned}

Below Equation (2), they also explain that $\vert \vert \mathbf{w} \vert \vert = g$, so we can rewrite the above to:

\begin{aligned} \nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w}}{\vert \vert \mathbf{w} \vert \vert} \cdot \frac{\mathbf{w}}{\vert \vert \mathbf{w} \vert \vert} \\ &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w} \mathbf{w}'}{\vert \vert \mathbf{w} \vert \vert^2} \\ \end{aligned}

In that last step, the $'$ symbol in $\mathbf{w}'$ denotes that we transpose that vector.

Finally, because we have that common $\frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L$ before and after the minus symbol, we can pull it out, multiplying it once by the identity matrix $\mathbf{I}$, and once by the remaining negated term. This is very similar to how you would simplify something like $a - ax$ to $a (1 - x)$ in "standard" algebra; the identity matrix $\mathbf{I}$ plays a very similar role here to the number $1$ in "standard" algebra:

\begin{aligned} \nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \left( \mathbf{I} - \frac{\mathbf{w} \mathbf{w}'}{\vert \vert \mathbf{w} \vert \vert^2} \right) \\ \end{aligned}

Note: in all of the above, I kind of neglected to pay attention to order of matrix/vector multiplications and their dimensions. Probably the $\nabla_{\mathbf{w}} L$ term should by now already have been moved to after those large brackets, instead of being before those brackets. In this case, it should work out fine regardless because $\mathbf{I}$ as well as $\mathbf{w} \mathbf{w}'$ are square and symmetric matrices. The only resulting difference is in whether you get a row or a column vector out at the end.

This can now finally be rewritten as the Equation you mentioned that you didn't understand yet, Equation (4) in the paper.

  • 2
    $\begingroup$ Hi Dennis, that's it! Thank you for the explanation, now I understand what was meant in the paper! It seems that I really need to work on my math and ml basics. Thank you again for the explanation! $\endgroup$
    – Grumpy C
    Commented Oct 3, 2018 at 18:25
  • 1
    $\begingroup$ @GrumpyC Well, that depends on what you want to do exactly. The ability to reproduce these kinds of math details is very different from the ability to implement the ideas and/or roughly understand the intuition. It's definitely useful to have a good basis in Linear Algebra though. $\endgroup$
    – Dennis Soemers
    Commented Oct 3, 2018 at 19:08
  • 1
    $\begingroup$ Hi Dennis, I think that both is necessary, intuition is probably enough for most of the practicioners, but I cannot but admire silent elegance of mathematics, in describing the world. Well I should properly read and practice! :D Thank you again! $\endgroup$
    – Grumpy C
    Commented Oct 3, 2018 at 21:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .