3

LSTMs or GRUs are computationally more effective than the standard RNNs because they explicitly attempt to address the vanishing and exploding gradient problems, which are numerical problems related to the vanishing or explosion of the values of the gradient vector (the vector that contains the partial derivatives of the loss function with respect to the ...


3

These newer RNNs (LSTMs and GRUs) have greater memory control, allowing previous values to persist or to be reset as necessary for many sequences of steps, avoiding "gradient decay" or eventual degradation of the values passed from step to step. LSTM and GRU networks make this memory control possible with memory blocks and structures called "gates" that pass ...


2

The second implementation looks more correct and inline with how Bidirectional is defined. Specifically, bidirectionality doen't change the forward/backward logic of either direction, and just merges (concat/sum/...) the outputs of forward/backward at a matching timestep t. You can check how Keras implements it here. There are distinct self.forward_layer ...


2

Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which links the size of the hidden state and the problem at hand. But, there are some rules of thumb like to start out with the size of the hidden state to be a power ...


1

It is not the sigmoid in particular. LSTMs and other memory-based recurrent networks are based on the idea of keeping an internal state that acts as a "canvas" in which the model can decide what to write (and thus keep in memory) and what to erase (and thus what to forget). Observe the top horizontal line in the image below. The line represents the ...


1

On the same problems, sometimes GRU is better, sometimes LSTM. In short, having more parameters (more "knobs") is not always a good thing. The training process needs to learn those parameters. There is a higher chance of over-fitting, amongst other problems. The parameters are assigned specific roles inside either GRU or LSTM, so if that role is less ...


Only top voted, non community-wiki answers of a minimum length are eligible