Linked Questions

97 votes
3 answers

What is self-supervised learning in machine learning?

What is self-supervised learning in machine learning? How is it different from supervised learning?
nbro's user avatar
  • 40.9k
3 votes
1 answer

Is it possible to use LLMs for regression tasks?

I want to use LLMs to predict edge weights in a graph based on attributes between two nodes. Is this even possible? If not, what would you recommend? I tried to look up uses of LLM in regression tasks,...
sharkeater123's user avatar
2 votes
2 answers

How does a LLM (transformer) pick words from its vocabulary?

I have a very rough understanding of the "attention/self attention" mechanism of transformer models and how this can be used to process a set of word vectors provided as an input/prompt to ...
MLBeginner's user avatar
3 votes
0 answers

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
Joe Black's user avatar
  • 181
6 votes
2 answers

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
Peyman's user avatar
  • 574
2 votes
1 answer

How can a transformer encoder attend to future tokens?

What does attending to future tokens mean? From my understanding, the transformer model works by inputting a prompt and predicting the next word in a sequence and this process just keeps repeating ...
Spike's user avatar
  • 21
0 votes
1 answer

How does Chat GPT encode a question?

Chat GPT is based on a decoder-only Transformer so it does not have an encoder. Given that, how is a user's question passed as input to Chat GPT's decoder? In a regular encoder-decoder architecture, ...
joan's user avatar
  • 1
0 votes
1 answer

what are the applications scenarios for prefix decoder LMs

Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only ...
Singh's user avatar
  • 1
2 votes
0 answers

How are weight matrices in attention learned?

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight ...
Ege Demir's user avatar
0 votes
1 answer

For a transformer decoder, how exactly are K, Q, and V for each decoding step?

For a transformer decoder, how exactly are K, Q, and V for each decoding step? Assume my input prompt is "today is a" (good day). At t= 0 (generation step 0): K, Q, and V are the projections ...
wrek's user avatar
  • 183
0 votes
2 answers

How the Q,K,V be calculated in multi-head attention

I want to understand the transformer architecture, so I start with self attention and I understand their mechanism, but when I pass to the multi-head attention I find some difficulties like how ...
1 vote
1 answer

How are the parts of GPT connected?

Reading Stephen Wolfram's explanation of ChatGPT, it sounds as if first you train a very powerful "autocomplete" function that doesn't know anything specifically about chatbots, and then you ...
ak0000's user avatar
  • 205
0 votes
0 answers

How are the transformer encoder outputs handled?

According to the Attention Is All You Need paper, the transformer's encoder portion is described as The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The ...
rkuang25's user avatar