# Does the paper “On the difficulty of training Recurrent Neural Networks” (2013) assume, falsely, that spectral radii are $\ge$ square matrix norms?

In section 2.1 the authors define $$\gamma$$ as the maximum possible value of the derivative of the activation function (e.g. 1 for tanh.) Then they have this to say:

We first prove that it is sufficient for $$\lambda_1 < \frac{1}{\gamma}$$, where $$\lambda_1$$ is the absolute value of the largest eigenvalue of the recurrent weight matrix $$W_{rec}$$, for the vanishing gradient problem to occur.

Then they use the submultiplicity ($$\|AB\| \le \|A\|\|B\|$$) of the 2-norm of the Jacobians to obtain the following inequality:

$$\forall x, \| \frac{\partial x_{k+1}}{\partial x_k} \| \le \| W_{rec}^\top \| \| diag(\sigma'(x_k))\| < \frac{1}{\gamma} \gamma < 1$$

Here

• $$x_k$$ is the pre-activated state vector of the RNN
• $$W_{rec}$$ is the weight matrix between states (i.e. $$x_k = W_{rec} \times \sigma(x_{k-1}) + b$$ )
• $$\sigma()$$ is the activation function for the state vector
• $$diag(v)$$ is the diagonal matrix version of a vector $$v$$

They appear to be either substituting the norm of the weight matrix $$\|W_{rec}^\top\|$$ for its largest eigenvalue $$|\lambda_1|$$ (eigenvalues are the same for transposes) or just assuming that this norm is less than or equal to the eigenvalue. This bothers me because the norm of a matrix is bounded below, not above, by this eigenvalue/spectral radius (see lemma 10 here and this math SE question)

They seem to assume that

$$\| W_{rec}^\top \| \le \lambda_1 < \frac{1}{\gamma}$$

But really

$$\| W_{rec}^\top \| \ge \lambda_1$$

We first prove that it is sufficient for $$\lambda_1 < \frac{1}{\gamma}$$, where $$\lambda_1$$ is the largest singular value of $$W_{rec}$$, for the vanishing gradient problem to occur.