13
votes
Accepted
What exactly is a hidden state in an LSTM and RNN?
This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.
Let's consider the following two input and output sequences
\begin{align}
X &...
- 1,036
3
votes
What exactly is a hidden state in an LSTM and RNN?
As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
- 1,409
3
votes
Why do we need both encoder and decoder in sequence to sequence prediction?
(Old question, I know...)
It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often.
Example for ...
- 351
3
votes
What exactly is a hidden state in an LSTM and RNN?
I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
- 31
3
votes
Accepted
Which situation will helpful using encoder or decoder or both in transformer model?
The original transformer paper presents the transformer as a model consisting of both encoder and decoder. However, many times you will see (or hear) people describing their model as a "...
- 612
2
votes
Accepted
Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?
I suggest to follow the official U-NET implementation.
To me, the second option ...
- 2,443
2
votes
Accepted
For a transformer decoder, how exactly are K, Q, and V for each decoding step?
(This type of) autoregressive LLM always works by predicting one next token based on a series of previous tokens. First you run the model with input "today is a" and the prediction is "...
- 932
2
votes
Accepted
How do temperature and repetition penalty interfere?
TL;DR: Temperature is applied after repetition penalty, so it smoothes out its effect.
They are basically independent hyper-parameters of the decoding, but applied after each other.
Higher temperature ...
- 381
2
votes
Accepted
Transformers: how does stacking work?
One encoder block of the transformer takes as input one tensor X and multiplies that by $W_Q$, $W_K$, $W_V$ to calculate $Q$, $K$, $V$ needed in self-attention.
After performing attention and feed-...
2
votes
What exactly is a hidden state in an LSTM and RNN?
The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step.
A simple ...
- 121
1
vote
Why encoders are required in Transformers
Transformers are based on encoder-decoder architectures for sequence processing, the point of this is to allow different input and output sequence lengths, and soft-attention was initially defined for ...
- 1,252
1
vote
Accepted
How does mixing and matching encoders and decoders work in image segmentation?
It's possible to mix and match all sorts of encoders and decoders. If the output of the encoder can be mapped to the input of the decoder, and a loss function can be backpropagated through the model, ...
- 2,083
1
vote
Accepted
What is a "mask" in the context o RNN-based encoders?
Masks in Recurrent Neural Networks are used to transform variable-length inputs to one general length. Therefore we use padding and masking together.
Padding: Usually we create a vector for every ...
- 126
1
vote
Seq2Seq model produces repeating words
The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This ...
- 2,426
Only top scored, non community-wiki answers of a minimum length are eligible