13
votes
Accepted
What exactly is a hidden state in an LSTM and RNN?
This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.
Let's consider the following two input and output sequences
\begin{align}
X &...
- 1,036
4
votes
What's the difference between content-based attention and dot-product attention?
The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor:
To illustrate why the dot products get large, assume that the components of $q$...
- 2,416
4
votes
How is Google Translate able to convert texts of different lengths?
Usually, in natural language processing (NLP), they are using Sequence to Sequence Learning (Seq2Seq) with Neural Networks, such as Recurrent Neural Networks or more recently the Transformer (you can ...
- 1,108
4
votes
Accepted
Why is it called a Seq2Seq model if the output is just a number?
The premise of your question is wrong. A model that goes from a sequence to a single prediction is simply NOT called a sequence to sequence to model.
The model type you are describing is called a ...
- 2,225
3
votes
Accepted
Seq2seq with RNNs, how is the training loop performed?
In seq2seq RNN training, we usually use a technique called "teacher forcing." With teacher forcing, the actual (ground truth) output word at each timestep is fed as input to the decoder in ...
3
votes
Can Reinforcement Learning be used to generate sequences?
One renowned example for the specified case is SeqGAN
Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by ...
- 1,731
3
votes
What exactly is a hidden state in an LSTM and RNN?
As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
- 1,409
3
votes
What exactly is a hidden state in an LSTM and RNN?
I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
- 31
2
votes
What exactly is a hidden state in an LSTM and RNN?
The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step.
A simple ...
- 121
2
votes
How does GPT like Decoder only conversational models distunguish the source of text?
A decoder-only conversational model, like GPT-3, generates text based on the context provided to it. It doesn't inherently "distinguish" the source of the text in the way humans might ...
- 1,583
1
vote
How to train a seq2seq model to rephrase input text following given rules
Data
As it seems like ChatGPT already works well, you could use ChatGPT to generate training data.
Models
I would look at finetuning an existing text paraphrasing model or abstractive summarization ...
- 654
1
vote
Is the decoder in a transformer Seq2Seq model non parallelizable?
During training, the decoder can be trained in parallel (and that's one of its advantage over LSTM) :
You input <s> I love you, and the decoder learns to ...
- 371
1
vote
How does Seq2Seq with attention actually use the attention (i.e. the context vector)?
Evidently you can only initialize it ($\vec{c_t}$) once
As I see it, $\vec{c_t}$ depends on $\vec{h}[1] \ldots \vec{h}[n]$ AND $\vec{s_{t-1}}$ (because $\alpha_i[t]$ depend on $\vec{s_{t-1}}$, and so ...
- 11
1
vote
Accepted
Does Seq2Seq decoder take a special vector or the weights of the last encoder cell as an output?
That drawing it's a bit oversimplified. Check this blog for a better explanation and implementation details. I'll refer to the image they have to answer:
the yellow boxes represent embedding layers, ...
- 5,183
1
vote
Seq2Seq model produces repeating words
The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This ...
- 2,416
1
vote
Do Seq2Seq models and the Bidirectional RNN do the same thing?
Seq2Seq and Bidirectional RNNs are not doing the same thing, at least in their classic form.
Seq2Seq models are used to generate a sequence from another sequence. Consider, for example, the ...
- 381
Only top scored, non community-wiki answers of a minimum length are eligible