14 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
Eka's user avatar
  • 1,066
5 votes

What's the difference between content-based attention and dot-product attention?

The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor: To illustrate why the dot products get large, assume that the components of $q$...
Kostya's user avatar
  • 2,524
4 votes
Accepted

Why is it called a Seq2Seq model if the output is just a number?

The premise of your question is wrong. A model that goes from a sequence to a single prediction is simply NOT called a sequence to sequence to model. The model type you are describing is called a ...
chessprogrammer's user avatar
4 votes

What are the differences between seq2seq and encoder-decoder architectures?

They are not the same, but they can overlap. An encoder-decoder architecture is composed of an encoder (which compresses the input) and a decoder (which decompresses the compressed input). A sequence-...
nbro's user avatar
  • 40.6k
4 votes

How is Google Translate able to convert texts of different lengths?

Usually, in natural language processing (NLP), they are using Sequence to Sequence Learning (Seq2Seq) with Neural Networks, such as Recurrent Neural Networks or more recently the Transformer (you can ...
razvanc92's user avatar
  • 1,128
3 votes

What exactly is a hidden state in an LSTM and RNN?

The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step. A simple ...
Burrito's user avatar
  • 141
3 votes

What exactly is a hidden state in an LSTM and RNN?

I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
ticiarai's user avatar
3 votes

What exactly is a hidden state in an LSTM and RNN?

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
ashenoy's user avatar
  • 1,409
3 votes

Can Reinforcement Learning be used to generate sequences?

One renowned example for the specified case is SeqGAN Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by ...
OmG's user avatar
  • 1,816
3 votes

What are the differences between seq2seq and encoder-decoder architectures?

Yes, you may have read tutorials or texts using interchangeably because of close relationships, but actually, there is a subtle distinction. Encoder-Decoder: It contains two main components Encoder ...
Hiren Namera's user avatar
3 votes
Accepted

Seq2seq with RNNs, how is the training loop performed?

In seq2seq RNN training, we usually use a technique called "teacher forcing." With teacher forcing, the actual (ground truth) output word at each timestep is fed as input to the decoder in ...
Hans-Peter Schrei's user avatar
2 votes

How does GPT like Decoder only conversational models distunguish the source of text?

A decoder-only conversational model, like GPT-3, generates text based on the context provided to it. It doesn't inherently "distinguish" the source of the text in the way humans might ...
DRV's user avatar
  • 1,683
1 vote

How to train a seq2seq model to rephrase input text following given rules

Data As it seems like ChatGPT already works well, you could use ChatGPT to generate training data. Models I would look at finetuning an existing text paraphrasing model or abstractive summarization ...
Alexander Wan's user avatar
1 vote

Is the decoder in a transformer Seq2Seq model non parallelizable?

During training, the decoder can be trained in parallel (and that's one of its advantage over LSTM) : You input <s> I love you, and the decoder learns to ...
Astariul's user avatar
  • 371
1 vote

How does Seq2Seq with attention actually use the attention (i.e. the context vector)?

Evidently you can only initialize it ($\vec{c_t}$) once As I see it, $\vec{c_t}$ depends on $\vec{h}[1] \ldots \vec{h}[n]$ AND $\vec{s_{t-1}}$ (because $\alpha_i[t]$ depend on $\vec{s_{t-1}}$, and so ...
Nathan's user avatar
  • 11
1 vote
Accepted

Does Seq2Seq decoder take a special vector or the weights of the last encoder cell as an output?

That drawing it's a bit oversimplified. Check this blog for a better explanation and implementation details. I'll refer to the image they have to answer: the yellow boxes represent embedding layers, ...
Edoardo Guerriero's user avatar
1 vote

Seq2Seq model produces repeating words

The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This ...
Kostya's user avatar
  • 2,524
1 vote

Do Seq2Seq models and the Bidirectional RNN do the same thing?

Seq2Seq and Bidirectional RNNs are not doing the same thing, at least in their classic form. Seq2Seq models are used to generate a sequence from another sequence. Consider, for example, the ...
hola's user avatar
  • 381

Only top scored, non community-wiki answers of a minimum length are eligible