14 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
Eka's user avatar
  • 1,066
6 votes
Accepted

Which situation will helpful using encoder or decoder or both in transformer model?

The original transformer paper presents the transformer as a model consisting of both encoder and decoder. However, many times you will see (or hear) people describing their model as a "...
pi-tau's user avatar
  • 797
4 votes

What are the differences between seq2seq and encoder-decoder architectures?

They are not the same, but they can overlap. An encoder-decoder architecture is composed of an encoder (which compresses the input) and a decoder (which decompresses the compressed input). A sequence-...
nbro's user avatar
  • 40.5k
3 votes

What are the differences between seq2seq and encoder-decoder architectures?

Yes, you may have read tutorials or texts using interchangeably because of close relationships, but actually, there is a subtle distinction. Encoder-Decoder: It contains two main components Encoder ...
Hiren Namera's user avatar
3 votes

What exactly is a hidden state in an LSTM and RNN?

The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step. A simple ...
Burrito's user avatar
  • 141
3 votes

What exactly is a hidden state in an LSTM and RNN?

I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
ticiarai's user avatar
3 votes

What exactly is a hidden state in an LSTM and RNN?

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
ashenoy's user avatar
  • 1,409
3 votes

Why do we need both encoder and decoder in sequence to sequence prediction?

(Old question, I know...) It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often. Example for ...
Mathias Müller's user avatar
2 votes
Accepted

Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?

I suggest to follow the official U-NET implementation. To me, the second option ...
Luca Anzalone's user avatar
2 votes
Accepted

For a transformer decoder, how exactly are K, Q, and V for each decoding step?

(This type of) autoregressive LLM always works by predicting one next token based on a series of previous tokens. First you run the model with input "today is a" and the prediction is "...
user253751's user avatar
2 votes
Accepted

How do temperature and repetition penalty interfere?

TL;DR: Temperature is applied after repetition penalty, so it smoothes out its effect. They are basically independent hyper-parameters of the decoding, but applied after each other. Higher temperature ...
Jindřich's user avatar
  • 391
2 votes
Accepted

Transformers: how does stacking work?

One encoder block of the transformer takes as input one tensor X and multiplies that by $W_Q$, $W_K$, $W_V$ to calculate $Q$, $K$, $V$ needed in self-attention. After performing attention and feed-...
Jakub Podolak's user avatar
2 votes
Accepted

Why do we do need compression in Semantic Segmentation?

There is never a 100% accurate theory, however it's been observed to be beneficial, however I would argue that is due to the following: you want to have a latent dimension, to learn the manifold ...
Alberto's user avatar
  • 1,885
2 votes

Why aren't encoders decoders trivial?

When an encoder outputs an embedded vector that is smaller than its input, the compression is lossy. The reversal process is not simple, it cannot be derived purely from analysis of input weights, but ...
Neil Slater's user avatar
  • 32.1k
2 votes

Transformers - how do the decoder attention input matrices look like, in terms of future tokens?

In addition to the answer by Cesar Ruiz: In the Transformer model, padding tokens are used to make all sequences the same length. These padding tokens are not actual data and are assigned a logit of <...
randomuser94786's user avatar
2 votes
Accepted

Transformers - how do the decoder attention input matrices look like, in terms of future tokens?

In the implementation of transformers, there are specific methods employed to address this issue. The attention mechanism initially establishes a context length, which refers to the number of tokens ...
Cesar Ruiz's user avatar
1 vote

Why do Transformer decoders use masked self attention when producing new tokens?

Its all about speed. During training: you use teacher forcing and you feed the entire target sequence $Y$ to the decoder (say of length $N$). You want the decoder to attend to ...
pi-tau's user avatar
  • 797
1 vote

Why can decoder-only transformers be so good at machine translation?

Docoder-only transformers are also trained on text pairs. You first pre-train on general text data. After that you fine tune with a standard two language dataset containing example translations. The ...
pi-tau's user avatar
  • 797
1 vote

Why encoders are required in Transformers

Transformers are based on encoder-decoder architectures for sequence processing, the point of this is to allow different input and output sequence lengths, and soft-attention was initially defined for ...
Dr. Snoopy's user avatar
  • 1,355
1 vote
Accepted

How does mixing and matching encoders and decoders work in image segmentation?

It's possible to mix and match all sorts of encoders and decoders. If the output of the encoder can be mapped to the input of the decoder, and a loss function can be backpropagated through the model, ...
Robin van Hoorn's user avatar
1 vote
Accepted

What is a "mask" in the context o RNN-based encoders?

Masks in Recurrent Neural Networks are used to transform variable-length inputs to one general length. Therefore we use padding and masking together. Padding: Usually we create a vector for every ...
technik's user avatar
  • 126
1 vote

Seq2Seq model produces repeating words

The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This ...
Kostya's user avatar
  • 2,524

Only top scored, non community-wiki answers of a minimum length are eligible