Hot answers tagged

7 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
user avatar
  • 946
4 votes

What's the difference between content-based attention and dot-product attention?

The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor: To illustrate why the dot products get large, assume that the components of $q$...
user avatar
  • 1,833
4 votes

How is Google Translate able to convert texts of different lengths?

Usually, in natural language processing (NLP), they are using Sequence to Sequence Learning (Seq2Seq) with Neural Networks, such as Recurrent Neural Networks or more recently the Transformer (you can ...
user avatar
  • 1,098
3 votes

What exactly is a hidden state in an LSTM and RNN?

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
user avatar
  • 1,369
2 votes

Can Reinforcement Learning be used to generate sequences?

One renowned example for the specified case is SeqGAN Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by ...
user avatar
  • 1,663
2 votes

What exactly is a hidden state in an LSTM and RNN?

I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
user avatar
1 vote
Accepted

Does Seq2Seq decoder take a special vector or the weights of the last encoder cell as an output?

That drawing it's a bit oversimplified. Check this blog for a better explanation and implementation details. I'll refer to the image they have to answer: the yellow boxes represent embedding layers, ...
user avatar
1 vote

Seq2Seq model produces repeating words

The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This ...
user avatar
  • 1,833
1 vote

Do Seq2Seq models and the Bidirectional RNN do the same thing?

Seq2Seq and Bidirectional RNNs are not doing the same thing, at least in their classic form. Seq2Seq models are used to generate a sequence from another sequence. Consider, for example, the ...
user avatar
  • 381
1 vote

What exactly is a hidden state in an LSTM and RNN?

The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step. A simple ...
user avatar
  • 111

Only top scored, non community-wiki answers of a minimum length are eligible