What is an identity recurrent neural network (IRNN)? What is the difference between an IRNN and RNN?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
An identity recurrent neural network (IRNN) is a vanilla recurrent neural network (as opposed to e.g. LSTMs) whose recurrent weight matrices are initialized with the identity matrix, the biases are initialized to zero, and the hidden units (or neurons) use the rectified linear unit (ReLU).
An IRNN can be trained more easily using gradient descent (as opposed to a vanilla RNN that is not an IRNN), given that it behaves similarly to an LSTM-based RNN, that is, an IRNN does not suffer (much) from the vanishing gradient problem.
The IRNN achieves a performance similar to LSTM-based RNNs in certain tasks, including the adding problem (a standard problem that is used to examine the power of recurrent models in learning long-term dependencies). In terms of architecture, the vanilla RNNs are much simpler than LSTM-based RNNs, so this is an advantage.
For more details, see the paper A Simple Way to Initialize Recurrent Networks of Rectified Linear Units (2015), by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton. See also this Keras implementation of the MNIST experiment described in the linked paper.