Skip to main content
8 votes
Accepted

Which situation will helpful using encoder or decoder or both in transformer model?

The original transformer paper presents the transformer as a model consisting of both encoder and decoder. However, many times you will see (or hear) people describing their model as a "...
pi-tau's user avatar
  • 815
8 votes
Accepted

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?

I got an answer to this question, probably a correct explanation. In decoder-only model architectures, the output of the model is a continuation of the model input. For example, input: I love apple [...
尹雅博's user avatar
7 votes

Why are embeddings added, not concatenated?

In high-dimensional spaces, the token embeddings and positional encodings can be thought of as forming two separate subspaces that are approximately orthogonal to each other. This is based on the ...
Abdur Rahman's user avatar
7 votes
Accepted

How can Transformers handle random sequences?

The stage of setting the available token encodings when training LLMs is very early on. It is before, and separate to the token prediction training (the core training process for the models). It is ...
Neil Slater's user avatar
  • 32.7k
6 votes

How do open source LLMs compare to GPT-4?

The remarkable performance of GPT 4 is due to the massive size of its architecture and the amount of data it was trained on, which costs a lot of money. Few organizations have the hardware resources ...
Brian O'Donnell's user avatar
6 votes
Accepted

Why different noise in GAN generate different images?

Let's take this apart. GANs stand for Generative Adversarial Networks. Your question is how GANs are generative (the G part of the name). For this we need to understand what they try to achieve. In ...
sfotiadis's user avatar
  • 291
5 votes

Can someone help me understand the intuition behind the query, key and value matrices in the transformer architecture?

A very very distant connection can be seen between the self-attention layer and the word2vec model. I think that this might be helpful to in order to gain more intution. Starting from the word2vec ...
pi-tau's user avatar
  • 815
4 votes

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

Assume that the query embeddings $Q$ and key embeddings $K$ have zero mean and unit std. Then for the variance of the attention score between any query and key we get: $$ \alpha = q_i k_j^T = \sum_{n=...
pi-tau's user avatar
  • 815
4 votes

What if we drop the causal mask in auto-regressive Transformer?

The purpose of the triangular causal mask in the attention mechanism is to enforce the autoregressive property of the model during training and inference. This property ensures that the model can only ...
Revolucion for Monica's user avatar
4 votes

How do open source LLMs compare to GPT-4?

How do open source LLMs compare to GPT-4? https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard has a leaderboard containing both open source LLMs and GPT-4 (and GPT-3.5-turbo): Model ⭐ ...
Franck Dernoncourt's user avatar
4 votes
Accepted

Why in Multi-Head Attention implementation should we use $3$ linear layers for Q, K, V instead of $3 * h$ layers?

It is just an optimization technique. If you have a vector $x$ of size $d$ and you want to multiply with $n$ different matrices $W_i$ of shape $d \times d_k$, then you could simply stack these ...
pi-tau's user avatar
  • 815
4 votes

Would AlphaZero perform better if made with transformers?

Yes! LeelaChessZero, an open-source re-implementation and continuation of AlphaZero, has been experimenting with this for a while now. Their strongest networks are currently transformers, not ...
KarelPeeters's user avatar
4 votes
Accepted

Is there a relationship between tokens and parameters in LLMs?

Transformers parameters count is invariant to the context window (which by definition can be infinite, though the $O(n^2)$ complexity might hurt) Consider that the context window that you see in the ...
Alberto's user avatar
  • 2,293
3 votes
Accepted

Can someone help me understand the intuition behind the query, key and value matrices in the transformer architecture?

Something I found helpful was "Transformers for Software Engineers" - unrolling the matrix multiplications into a funky functional program which maps over vectors. We can follow this ...
llllvvuu's user avatar
  • 146
3 votes
Accepted

Why does averaging attention-weighted positions reduce the effective resolution in transformers?

I think that the idea of "reduced effective resolution" from averaging is best understood through seeing how the proposed multi-head attention architecture fixes the issue. Specifically, ...
Andrew Du's user avatar
3 votes

Why is dot-product and not Euclidean distance used for attention?

Because it's a good fit, pretty much Yes, you can use any "inverse distance measure", however, scalar product paired with softmax is a out-of-the-box good fit, as two vectors the less they ...
Alberto's user avatar
  • 2,293
3 votes
Accepted

Aren't context lengths for transformers an artificial restriction?

Yes, you have the right idea. There's been a lot of work recently regarding extending the context-length of existing models, mostly looking at the Llama family of models. You should check out this ...
Alexander Wan's user avatar
3 votes

Does a decoder in transformer model generate output embeddings like the following?

The encoder-decoder architecture is typically employed to work with two different sequences: the input sequence and the output sequence. The presence of both depends on the task; for machine ...
Nicola Fanelli's user avatar
3 votes

Why different noise in GAN generate different images?

To further address you comment question, it's GAN's generator network's deep learning ability consisting of multiple layers of nonlinear transformations (e.g., convolutional layers, transposed ...
cinch's user avatar
  • 2,277
2 votes

Why do the values in the cross attentional mechanism within a transformer come from the encoder and not from the decoder?

The idea of the cross-attention layer is to transform the input words to output words. The Decoder provides context of which input words should we pay attention to next based on the already decoded ...
Abhishek's user avatar
  • 121
2 votes
Accepted

How can an decoder-only transformer be used for document embedding?

The GPT models (as manifested using the decoder block in the original Transformer architecture) are not generating the embedding. However, the weights from the GPT are being used as the initial ...
sghael's user avatar
  • 156
2 votes
Accepted

Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?

The short answer: Very unlikely. The extended answer: If you fine-tune a model, it becomes specialized for the type of data you fine-tune it on but you trade in some of its generalization capabilities....
emely_pi's user avatar
  • 277
2 votes
Accepted

How is the padding mask incorporated in the attention formula?

Entries of an attention mask are typically either $0$ or $-\infty$. So, adding such a mask gives either the original entry of $QK^T$ or $-\infty$. The issue with entrywise multiplication with a binary ...
Venna Banana's user avatar
2 votes

Modern graduate-level machine learning books with focus on generative models

Berkeley CS294-158 is a graduate-level course on deep unsupervised learning. They cover a lot of architectures used in modern generative modeling. They have recorded lectures and slides online. ...
Alexander Wan's user avatar
2 votes

How does GPT like Decoder only conversational models distunguish the source of text?

A decoder-only conversational model, like GPT-3, generates text based on the context provided to it. It doesn't inherently "distinguish" the source of the text in the way humans might ...
DRV's user avatar
  • 1,703
2 votes
Accepted

Why is there a shared matrix W in graph attention networks instead of the query-key-value trio like in regular transformers?

To my understanding, there isn't any theoretical reason why the query, key and values weights are absent. I feel that the difference may lie in the way the additive attention is calculated vs the dot-...
Cesar Ruiz's user avatar
2 votes

Time Series Classification using Transformer Encoder

If you know for a fact that you will always have 64 points in the time series, then there is no advantage of using a transformer over a MLP. The advantage of the transformer is it can batch process ...
Karl's user avatar
  • 206
2 votes

From where do the Encoders in Transformers gets Input Embedding from?

Either by building embeddings yourself or loading pretrained embeddings. For building yourself, this is typically done with an auto-regressive model. It can be as simple as creating numeric ...
David Hoelzer's user avatar
2 votes
Accepted

Why is it called multi-headed attention?

The original paper "Attention is all you need" mentions the following. " Instead of performing a single attention function with $ d_{model} $-dimensional keys, values and queries, we ...
Sathishkumar Thirumalai's user avatar
2 votes
Accepted

Transformers - how do the decoder attention input matrices look like, in terms of future tokens?

In the implementation of transformers, there are specific methods employed to address this issue. The attention mechanism initially establishes a context length, which refers to the number of tokens ...
Cesar Ruiz's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible