3 votes

Why are GRU and LSTM better than standard RNNs?

LSTMs or GRUs are computationally more effective than the standard RNNs because they explicitly attempt to address the vanishing and exploding gradient problems, which are numerical problems related ...
user avatar
  • 34.3k
3 votes
Accepted

Why are GRU and LSTM better than standard RNNs?

These newer RNNs (LSTMs and GRUs) have greater memory control, allowing previous values to persist or to be reset as necessary for many sequences of steps, avoiding "gradient decay" or eventual ...
user avatar
2 votes
Accepted

Inner working of Bidirectional RNNs

The second implementation looks more correct and inline with how Bidirectional is defined. Specifically, bidirectionality doen't change the forward/backward logic of either direction, and just merges (...
user avatar
  • 266
2 votes
Accepted

How do I choose the size of the hidden state of a GRU?

Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which ...
user avatar
  • 542
1 vote

Why does validation accuracy stop rising so soon?

Since all networks' accuracy goes close to 100%, I would argue that all of the models are capable of learning this task. But the first two models are somewhat overfitting, since the validation ...
user avatar
  • 345
1 vote

Is my intuition about RNN wrong?

I got this batch-based GRU to work: ...
user avatar
  • 345
1 vote
Accepted

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

It is not the sigmoid in particular. LSTMs and other memory-based recurrent networks are based on the idea of keeping an internal state that acts as a "canvas" in which the model can decide ...
user avatar
  • 26
1 vote
Accepted

What's the difference between LSTM and GRU?

On the same problems, sometimes GRU is better, sometimes LSTM. In short, having more parameters (more "knobs") is not always a good thing. The training process needs to learn those parameters. There ...
user avatar
  • 24.5k

Only top scored, non community-wiki answers of a minimum length are eligible