Questions tagged [encoder-decoder]
The encoder-decoder tag has no usage guidance.
29
questions
0
votes
0
answers
8
views
An issue about the Decoder in seq2seq(rnn)
Here is a confusion about the decoder in seq2seq.
In each time-step in decoder, there are two outputs: 1.output 2.hidden.
and this hidden state is used as the next input hidden state.
this output is ...
- 1
1
vote
0
answers
36
views
What’s more efficient in multihead attention: multiply QKV by $W_i$ then split or linearly project QKV $h$ times into dimensions $d_k$?
I’m looking to bridge two implementations of multihead attention.
Approach 1: Multiply and Split
Each of the queries, keys, and values is multiplied by a separate square weight matrix of size (...
- 11
1
vote
0
answers
76
views
How to Interpret Cross Attention
I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. ...
0
votes
0
answers
16
views
Transformers Decoder Inputs (Keys and Values)
I am trying to better understand the encoder-decoder transformer architecture.
I understand the high-level intuition behind the concept of keys, queries and values. For example, in the context of CV, ...
1
vote
1
answer
319
views
Why do Transformer decoders use masked self attention when producing new tokens?
I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
1
vote
1
answer
67
views
Why aren't encoders decoders trivial?
If you have an encoder decoder with 10 input neurons for X then 3 hidden in one layer then another 10 in the output which are the same X is it not trivial to set the weights whatever you want and w1 ...
- 11
0
votes
0
answers
21
views
Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?
here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
0
votes
2
answers
109
views
Transformers - how do the decoder attention input matrices look like, in terms of future tokens?
I have a question regarding the original transformer implementation (as in "Attention is all you need").
Assuming I want to translate English to German.
In the Decoder part, in the self-attention ...
- 15
0
votes
1
answer
794
views
How does Chat GPT encode a question?
Chat GPT is based on a decoder-only Transformer so it does not have an encoder. Given that, how is a user's question passed as input to Chat GPT's decoder? In a regular encoder-decoder architecture, ...
- 1
4
votes
2
answers
2k
views
What are the differences between seq2seq and encoder-decoder architectures?
I've read many tutorials online that use both words interchangeably. When I search and find that they are the same, why not just use one word since they have the same definition?
user78615
1
vote
1
answer
46
views
Why do we do need compression in Semantic Segmentation?
When doing semantic segmentation, we often make use of FCN, which can be thought of in two parts: an encoder and decoder. As I understand, the encoder compresses the image into a spatially small, but ...
- 113
1
vote
0
answers
65
views
What is the input to an encoder-decoder transformer in next word prediction task?
I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
- 111
0
votes
2
answers
210
views
Why encoders are required in Transformers
In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....).
I mean even a decoder also has the same ...
- 101
5
votes
1
answer
3k
views
Which situation will helpful using encoder or decoder or both in transformer model?
I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.
The overall form of a transformer consists of an ...
- 67
1
vote
1
answer
852
views
Why can decoder-only transformers be so good at machine translation?
In my understanding encoder-decoder transformers for translation are trained with sentence or text pairs. How can it be explained in simple (high-level) terms that decoder-only transformers (e.g. GPT) ...
0
votes
1
answer
758
views
Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?
Context
I'm investigating the UNet architecture for a little while now. After investigating the structure of the official UNet architecture as proposed in the official paper I noticed a recurrent ...
0
votes
1
answer
184
views
For a transformer decoder, how exactly are K, Q, and V for each decoding step?
For a transformer decoder, how exactly are K, Q, and V for each decoding step?
Assume my input prompt is "today is a" (good day).
At t= 0 (generation step 0):
K, Q, and V are the projections ...
- 183
2
votes
0
answers
32
views
Combining GANs and NLP for AI-Based Programming: Generating Input-Output Templates for Computer Functions
I would like to combine GANs and NLP to create a system that can take an input and generate an appropriate output. For example, ...
- 21
0
votes
1
answer
5k
views
How do temperature and repetition penalty interfere?
I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature?
For example, does ...
- 103
1
vote
1
answer
1k
views
Transformers: how does stacking work? [closed]
An Encoder has as inputs : Q,K,V, but has single output i.e. 3 vs 1
How do you stack those ?
Is there more detailed diagram ?
- 113
0
votes
1
answer
285
views
How does mixing and matching encoders and decoders work in image segmentation?
I had a conceptual questions regarding architectures. I am using this git hub repository that allows one to quickly put together a segmentation pipeline. In reading the readme one thing that has me ...
- 103
1
vote
0
answers
397
views
Left-to-Right vs Encoder-decoder Models
Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS)
Left-to-Right:
Auto-regressive, Left-to-right models, predict the ...
- 39
1
vote
1
answer
612
views
How to Train a Decoder for Pre-trained BERT Transformer-Encoder?
Context:
I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
- 11
0
votes
1
answer
291
views
What is a "mask" in the context o RNN-based encoders?
While reading source code related to RNN encoders, I've come across the term mask as input to the encoder. What exactly is it?
- 3,960
3
votes
0
answers
2k
views
What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining.
I'm trying to understand the shapes used during decoder (both self-...
- 181
0
votes
1
answer
1k
views
Seq2Seq model produces repeating words
My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
1
vote
1
answer
266
views
How is the transformers' output matrix size arrived at?
In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
- 111
17
votes
4
answers
31k
views
What exactly is a hidden state in an LSTM and RNN?
I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
- 805
4
votes
1
answer
2k
views
Why do we need both encoder and decoder in sequence to sequence prediction?
Why do we need both encoder and decoder in sequence to sequence prediction?
We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...
- 61