I was recently brushing up on my deep-learning basics and came back to RNNs. LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at the time also interested in image processing and the ResNet model, so I immediately thought of incorporating the residual connection into RNNs. The idea is to connect the hidden states after each block (skipping the network and activations) to allow for better training of the past blocks. Given that nobody has done this yet, I feel the idea is somehow flawed and not as genius as my mind thinks. Why would/wouldn't this work? Where did I make a mistake?


2 Answers 2


In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. There are also other architectures than ResNet that uses residuals, but add gate layers to guide what is passed on.

The main thing your idea would bring is likely speed, rather than accuracy. And that is not always a bad thing.

Here are some papers where the author uses a residuals in combination with LSTM:
Residual LSTM


Your idea is exactly the idea behind state-space models. They have a linear "residual" connection from previous hidden states, skipping activations. In fact, it works very well! I'd recommend reading the Mamba paper. They get comparable results to "transformers" while being several times more efficient, and fixing the context-length issue.

There are two major issues with LSTMs, and in general RNNs, that make training slow:

  1. Vanishing gradients (as you mentioned), and
  2. Only being able to train one token in a sequence at once.

Because state-space models just have a linear "residual" connection, they can train an entire sequence of tokens at once with a parallel scan algorithm (similar to the prefix sum algorithm). Note that this ability to crunch huge amounts of data is what allowed "transformers" to take off.

The issue you mentioned is actually a little broader: if you run an RNN too long the outputs will either vanish or explode. You can imagine a purely linear model, $x\mapsto Ax$, which after $n$ steps will become $A^{n}x$. If the eigenvalues are too big, things explode, while if they are too small, $A^{n}x\to 0$. The authors of Mamba considered forcing $A$ to be unitary (eigenvalues have magnitude exactly one, see section B.3), and mention that other authors have explored it in more depth to resolve this issue.


You must log in to answer this question.