Questions tagged [encoder-decoder]
The encoder-decoder tag has no usage guidance.
19
questions
1
vote
1
answer
32
views
Why do we do need compression in Semantic Segmentation?
When doing semantic segmentation, we often make use of FCN, which can be thought of in two parts: an encoder and decoder. As I understand, the encoder compresses the image into a spatially small, but ...
- 113
1
vote
0
answers
20
views
What is the input to an encoder-decoder transformer in next word prediction task?
I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
- 111
0
votes
2
answers
70
views
Why encoders are required in Transformers
In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....).
I mean even a decoder also has the same ...
- 1
1
vote
1
answer
698
views
Which situation will helpful using encoder or decoder or both in transformer model?
I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.
The overall form of a transformer consists of an ...
- 27
1
vote
1
answer
286
views
Why can decoder-only transformers be so good at machine translation?
In my understanding encoder-decoder transformers for translation are trained with sentence or text pairs. How can it be explained in simple (high-level) terms that decoder-only transformers (e.g. GPT) ...
0
votes
1
answer
243
views
Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?
Context
I'm investigating the UNet architecture for a little while now. After investigating the structure of the official UNet architecture as proposed in the official paper I noticed a recurrent ...
0
votes
1
answer
88
views
For a transformer decoder, how exactly are K, Q, and V for each decoding step?
For a transformer decoder, how exactly are K, Q, and V for each decoding step?
Assume my input prompt is "today is a" (good day).
At t= 0 (generation step 0):
K, Q, and V are the projections ...
- 183
2
votes
0
answers
27
views
Combining GANs and NLP for AI-Based Programming: Generating Input-Output Templates for Computer Functions
I would like to combine GANs and NLP to create a system that can take an input and generate an appropriate output. For example, ...
- 21
0
votes
1
answer
3k
views
How do temperature and repetition penalty interfere?
I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature?
For example, does ...
- 103
1
vote
1
answer
608
views
Transformers: how does stacking work? [closed]
An Encoder has as inputs : Q,K,V, but has single output i.e. 3 vs 1
How do you stack those ?
Is there more detailed diagram ?
- 113
0
votes
1
answer
103
views
How does mixing and matching encoders and decoders work in image segmentation?
I had a conceptual questions regarding architectures. I am using this git hub repository that allows one to quickly put together a segmentation pipeline. In reading the readme one thing that has me ...
- 103
1
vote
0
answers
300
views
Left-to-Right vs Encoder-decoder Models
Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS)
Left-to-Right:
Auto-regressive, Left-to-right models, predict the ...
- 39
1
vote
1
answer
311
views
How to Train a Decoder for Pre-trained BERT Transformer-Encoder?
Context:
I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
- 11
0
votes
1
answer
194
views
What is a "mask" in the context o RNN-based encoders?
While reading source code related to RNN encoders, I've come across the term mask as input to the encoder. What exactly is it?
- 3,612
3
votes
0
answers
2k
views
What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining.
I'm trying to understand the shapes used during decoder (both self-...
- 181
0
votes
1
answer
1k
views
Seq2Seq model produces repeating words
My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
1
vote
1
answer
211
views
How is the transformers' output matrix size arrived at?
In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
- 111
13
votes
4
answers
26k
views
What exactly is a hidden state in an LSTM and RNN?
I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
- 737
4
votes
1
answer
2k
views
Why do we need both encoder and decoder in sequence to sequence prediction?
Why do we need both encoder and decoder in sequence to sequence prediction?
We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...
- 61