Questions tagged [encoder-decoder]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
32 views

Why do we do need compression in Semantic Segmentation?

When doing semantic segmentation, we often make use of FCN, which can be thought of in two parts: an encoder and decoder. As I understand, the encoder compresses the image into a spatially small, but ...
Dude156's user avatar
  • 113
1 vote
0 answers
20 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
mehsheenman's user avatar
0 votes
2 answers
70 views

Why encoders are required in Transformers

In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....). I mean even a decoder also has the same ...
Swastik's user avatar
1 vote
1 answer
698 views

Which situation will helpful using encoder or decoder or both in transformer model?

I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer. The overall form of a transformer consists of an ...
Yang's user avatar
  • 27
1 vote
1 answer
286 views

Why can decoder-only transformers be so good at machine translation?

In my understanding encoder-decoder transformers for translation are trained with sentence or text pairs. How can it be explained in simple (high-level) terms that decoder-only transformers (e.g. GPT) ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
243 views

Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?

Context I'm investigating the UNet architecture for a little while now. After investigating the structure of the official UNet architecture as proposed in the official paper I noticed a recurrent ...
timu vlad's user avatar
0 votes
1 answer
88 views

For a transformer decoder, how exactly are K, Q, and V for each decoding step?

For a transformer decoder, how exactly are K, Q, and V for each decoding step? Assume my input prompt is "today is a" (good day). At t= 0 (generation step 0): K, Q, and V are the projections ...
wrek's user avatar
  • 183
2 votes
0 answers
27 views

Combining GANs and NLP for AI-Based Programming: Generating Input-Output Templates for Computer Functions

I would like to combine GANs and NLP to create a system that can take an input and generate an appropriate output. For example, ...
Doğuş Deniz's user avatar
0 votes
1 answer
3k views

How do temperature and repetition penalty interfere?

I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature? For example, does ...
Corbin's user avatar
  • 103
1 vote
1 answer
608 views

Transformers: how does stacking work? [closed]

An Encoder has as inputs : Q,K,V, but has single output i.e. 3 vs 1 How do you stack those ? Is there more detailed diagram ?
sten's user avatar
  • 113
0 votes
1 answer
103 views

How does mixing and matching encoders and decoders work in image segmentation?

I had a conceptual questions regarding architectures. I am using this git hub repository that allows one to quickly put together a segmentation pipeline. In reading the readme one thing that has me ...
TheCodeNovice's user avatar
1 vote
0 answers
300 views

Left-to-Right vs Encoder-decoder Models

Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS) Left-to-Right: Auto-regressive, Left-to-right models, predict the ...
keyboardAnt's user avatar
1 vote
1 answer
311 views

How to Train a Decoder for Pre-trained BERT Transformer-Encoder?

Context: I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
nesquick's user avatar
0 votes
1 answer
194 views

What is a "mask" in the context o RNN-based encoders?

While reading source code related to RNN encoders, I've come across the term mask as input to the encoder. What exactly is it?
hanugm's user avatar
  • 3,612
3 votes
0 answers
2k views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
Joe Black's user avatar
  • 181
0 votes
1 answer
1k views

Seq2Seq model produces repeating words

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
Cheleeger Ken's user avatar
1 vote
1 answer
211 views

How is the transformers' output matrix size arrived at?

In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
Alien's user avatar
  • 111
13 votes
4 answers
26k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
user8714896's user avatar
4 votes
1 answer
2k views

Why do we need both encoder and decoder in sequence to sequence prediction?

Why do we need both encoder and decoder in sequence to sequence prediction? We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...
greensquare's user avatar