7 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
user avatar
  • 946
3 votes

What exactly is a hidden state in an LSTM and RNN?

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
user avatar
  • 1,359
2 votes

What exactly is a hidden state in an LSTM and RNN?

I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
user avatar
2 votes

Why do we need both encoder and decoder in sequence to sequence prediction?

(Old question, I know...) It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often. Example for ...
user avatar
1 vote
Accepted

What is a "mask" in the context o RNN-based encoders?

Masks in Recurrent Neural Networks are used to transform variable-length inputs to one general length. Therefore we use padding and masking together. Padding: Usually we create a vector for every ...
user avatar
  • 26
1 vote

Seq2Seq model produces repeating words

The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This ...
user avatar
  • 1,833
1 vote

What exactly is a hidden state in an LSTM and RNN?

The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step. A simple ...
user avatar
  • 111

Only top scored, non community-wiki answers of a minimum length are eligible