Rank of gradient-of-loss with respect to layer weights in an MLP

Question

The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3:

The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch elements $\nabla_{w_l}L = \sum_{i=1}^B \delta_{l+1}^i {u_l^i}^T$, where $\delta_{l+1}^i$ is the gradient of the loss for batch element $i$ with respect to the preactivations at layer $l + 1$, and ${u_l^i}^T$ are the inputs to layer $l$ for batch element i.

Suppose that we have an MLP with $k$ hidden layers (every hidden layer is followed by an activation function). Then the weight matrices will be $W_1, W_2, \dots, W_k$ (plus the biases, but they are irrelevant for now), and their sizes will be $(D_1, D), (D_2, D_1), \dots (D_k, D_{k-1})$ correspondingly, where $D$ is the number of input features.

Therefore, hidden layer $l$ has a weight matrix $W_l$ of size $(D_l, D_{l-1})$. Its gradient wrt the loss (for 1 batch element), $\frac{\partial L}{\partial W_l}$, will also be a matrix of size $(D_l, D_{l-1})$.

So if I understand correctly, the authors of the paper are claiming that $\frac{\partial L}{\partial W_l}$ is a rank-1 matrix? That is, every row (or column) can be expressed as a linear combination of 1 only row (or column)? If yes, why? How?

Answer 1

I didn't find the reference in the Goodfellow deep learning book. But here is how I derived it. Let all vectors be column vectors.

The basic claim is that $\nabla_W F(u^\top W) = \nabla F(v^\top) u^\top$, where $v^\top := u^\top W$.

Consider the simplest function $F(v^\top) := v^\top x$. We can compute $\nabla_W (u^\top W x)$ either by looking up matrix differentiation rules, or by hand:

$$\partial_{W_{i,j}} u^\top W x = x_i u_j.$$ This shows that $\nabla_W (u^\top W x) = x u^\top$.

Finally convince yourself that general $F$ behaves the same way: $$\nabla_W F(u^\top W) = \nabla_W (u^\top W \nabla F(v^\top)) = \nabla F(v^\top) u^\top,$$ since the post-multiplication by $\nabla F(v^\top)$ spans all linear maps taking $u^\top W$ to $\mathbb{R}$.