In section 2.1 the authors define $\gamma$ as the maximum possible value of the derivative of the activation function (e.g. 1 for tanh.) Then they have this to say:
We first prove that it is sufficient for $\lambda_1 < \frac{1}{\gamma}$, where $\lambda_1$ is the absolute value of the largest eigenvalue of the recurrent weight matrix $W_{rec}$, for the vanishing gradient problem to occur.
Then they use the submultiplicity ($\|AB\| \le \|A\|\|B\|$) of the 2-norm of the Jacobians to obtain the following inequality:
$$ \forall x, \| \frac{\partial x_{k+1}}{\partial x_k} \| \le \| W_{rec}^\top \| \| diag(\sigma'(x_k))\| < \frac{1}{\gamma} \gamma < 1 $$
Here
- $x_k$ is the pre-activated state vector of the RNN
- $W_{rec}$ is the weight matrix between states (i.e. $x_k = W_{rec} \times \sigma(x_{k-1}) + b$ )
- $\sigma()$ is the activation function for the state vector
- $diag(v)$ is the diagonal matrix version of a vector $v$
They appear to be either substituting the norm of the weight matrix $\|W_{rec}^\top\|$ for its largest eigenvalue $|\lambda_1|$ (eigenvalues are the same for transposes) or just assuming that this norm is less than or equal to the eigenvalue. This bothers me because the norm of a matrix is bounded below, not above, by this eigenvalue/spectral radius (see lemma 10 here and this math SE question)
They seem to assume that
$$\| W_{rec}^\top \| \le \lambda_1 < \frac{1}{\gamma} $$
But really
$$ \| W_{rec}^\top \| \ge \lambda_1 $$