Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [encoder-decoder]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

What’s more efficient in multihead attention: multiply QKV by W_i then split or linearly project QKV h times into dimensions d_k?

I’m looking to bridge two implementations of multihead attention. Approach 1: Multiply and Split Each of the queries, keys, and values is multiplied by a separate square weight matrix of size (...
marcocamilo's user avatar
1 vote
0 answers
22 views

How to Interpret Cross Attention

I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. ...
Kiran Manicka's user avatar
0 votes
0 answers
11 views

Transformers Decoder Inputs (Keys and Values)

I am trying to better understand the encoder-decoder transformer architecture. I understand the high-level intuition behind the concept of keys, queries and values. For example, in the context of CV, ...
user2661372's user avatar
1 vote
1 answer
104 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
Kiran Manicka's user avatar
1 vote
1 answer
66 views

Why aren't encoders decoders trivial?

If you have an encoder decoder with 10 input neurons for X then 3 hidden in one layer then another 10 in the output which are the same X is it not trivial to set the weights whatever you want and w1 ...
J_Bake's user avatar
  • 11
0 votes
0 answers
20 views

Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?

here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
Jeremy Kenn's user avatar
0 votes
2 answers
105 views

Transformers - how do the decoder attention input matrices look like, in terms of future tokens?

I have a question regarding the original transformer implementation (as in "Attention is all you need"). Assuming I want to translate English to German. In the Decoder part, in the self-attention ...
Ben Lahav's user avatar
0 votes
1 answer
534 views

How does Chat GPT encode a question?

Chat GPT is based on a decoder-only Transformer so it does not have an encoder. Given that, how is a user's question passed as input to Chat GPT's decoder? In a regular encoder-decoder architecture, ...
joan's user avatar
  • 1
4 votes
2 answers
2k views

What are the differences between seq2seq and encoder-decoder architectures?

I've read many tutorials online that use both words interchangeably. When I search and find that they are the same, why not just use one word since they have the same definition?
user avatar
1 vote
1 answer
46 views

Why do we do need compression in Semantic Segmentation?

When doing semantic segmentation, we often make use of FCN, which can be thought of in two parts: an encoder and decoder. As I understand, the encoder compresses the image into a spatially small, but ...
Dude156's user avatar
  • 113
1 vote
0 answers
53 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
thevoid's user avatar
  • 111
0 votes
2 answers
145 views

Why encoders are required in Transformers

In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....). I mean even a decoder also has the same ...
Swastik's user avatar
  • 101
4 votes
1 answer
3k views

Which situation will helpful using encoder or decoder or both in transformer model?

I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer. The overall form of a transformer consists of an ...
Yang's user avatar
  • 57
1 vote
1 answer
648 views

Why can decoder-only transformers be so good at machine translation?

In my understanding encoder-decoder transformers for translation are trained with sentence or text pairs. How can it be explained in simple (high-level) terms that decoder-only transformers (e.g. GPT) ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
594 views

Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?

Context I'm investigating the UNet architecture for a little while now. After investigating the structure of the official UNet architecture as proposed in the official paper I noticed a recurrent ...
timu vlad's user avatar
0 votes
1 answer
142 views

For a transformer decoder, how exactly are K, Q, and V for each decoding step?

For a transformer decoder, how exactly are K, Q, and V for each decoding step? Assume my input prompt is "today is a" (good day). At t= 0 (generation step 0): K, Q, and V are the projections ...
wrek's user avatar
  • 183
2 votes
0 answers
30 views

Combining GANs and NLP for AI-Based Programming: Generating Input-Output Templates for Computer Functions

I would like to combine GANs and NLP to create a system that can take an input and generate an appropriate output. For example, ...
Doğuş Deniz's user avatar
0 votes
1 answer
5k views

How do temperature and repetition penalty interfere?

I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature? For example, does ...
Corbin's user avatar
  • 103
1 vote
1 answer
905 views

Transformers: how does stacking work? [closed]

An Encoder has as inputs : Q,K,V, but has single output i.e. 3 vs 1 How do you stack those ? Is there more detailed diagram ?
sten's user avatar
  • 113
0 votes
1 answer
216 views

How does mixing and matching encoders and decoders work in image segmentation?

I had a conceptual questions regarding architectures. I am using this git hub repository that allows one to quickly put together a segmentation pipeline. In reading the readme one thing that has me ...
TheCodeNovice's user avatar
1 vote
0 answers
364 views

Left-to-Right vs Encoder-decoder Models

Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS) Left-to-Right: Auto-regressive, Left-to-right models, predict the ...
keyboardAnt's user avatar
1 vote
1 answer
498 views

How to Train a Decoder for Pre-trained BERT Transformer-Encoder?

Context: I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
node_env's user avatar
0 votes
1 answer
257 views

What is a "mask" in the context o RNN-based encoders?

While reading source code related to RNN encoders, I've come across the term mask as input to the encoder. What exactly is it?
hanugm's user avatar
  • 3,890
3 votes
0 answers
2k views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
Joe Black's user avatar
  • 181
0 votes
1 answer
1k views

Seq2Seq model produces repeating words

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
Cheleeger Ken's user avatar
1 vote
1 answer
246 views

How is the transformers' output matrix size arrived at?

In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
Alien's user avatar
  • 111
16 votes
4 answers
29k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
user8714896's user avatar
4 votes
1 answer
2k views

Why do we need both encoder and decoder in sequence to sequence prediction?

Why do we need both encoder and decoder in sequence to sequence prediction? We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...
greensquare's user avatar