Does the paper "On the difficulty of training Recurrent Neural Networks" (2013) assume, falsely, that spectral radii are $\ge$ square matrix norms?

Question

(link to paper in arxiv)

In section 2.1 the authors define $\gamma$ as the maximum possible value of the derivative of the activation function (e.g. 1 for tanh.) Then they have this to say:

We first prove that it is sufficient for $\lambda_1 < \frac{1}{\gamma}$, where $\lambda_1$ is the absolute value of the largest eigenvalue of the recurrent weight matrix $W_{rec}$, for the vanishing gradient problem to occur.

Then they use the submultiplicity ($\|AB\| \le \|A\|\|B\|$) of the 2-norm of the Jacobians to obtain the following inequality:

$$ \forall x, \| \frac{\partial x_{k+1}}{\partial x_k} \| \le \| W_{rec}^\top \| \| diag(\sigma'(x_k))\| < \frac{1}{\gamma} \gamma < 1 $$

Here

• $x_k$ is the pre-activated state vector of the RNN
• $W_{rec}$ is the weight matrix between states (i.e. $x_k = W_{rec} \times \sigma(x_{k-1}) + b$ )
• $\sigma()$ is the activation function for the state vector
• $diag(v)$ is the diagonal matrix version of a vector $v$

They appear to be either substituting the norm of the weight matrix $\|W_{rec}^\top\|$ for its largest eigenvalue $|\lambda_1|$ (eigenvalues are the same for transposes) or just assuming that this norm is less than or equal to the eigenvalue. This bothers me because the norm of a matrix is bounded below, not above, by this eigenvalue/spectral radius (see lemma 10 here and this math SE question)

They seem to assume that

$$\| W_{rec}^\top \| \le \lambda_1 < \frac{1}{\gamma} $$

But really

$$ \| W_{rec}^\top \| \ge \lambda_1 $$

Answer 1

It is an error. But it is also not the in final version of the paper (arxiv.) The final version of the paper can be found here where they replace "absolute value of the largest eigenvalue" with "largest singular value".

We first prove that it is sufficient for $\lambda_1 < \frac{1}{\gamma}$, where $\lambda_1$ is the largest singular value of $W_{rec}$, for the vanishing gradient problem to occur.