Skip to main content

New answers tagged

2 votes

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

Your idea is exactly the idea behind state-space models. They have a linear "residual" connection from previous hidden states, skipping activations. In fact, it works very well! I'd ...
programjames's user avatar
3 votes
Accepted

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. ...
Victor Björkgren's user avatar

Top 50 recent answers are included