Linked Questions

97 votes
3 answers
86k views

What is self-supervised learning in machine learning?

What is self-supervised learning in machine learning? How is it different from supervised learning?
nbro's user avatar
  • 40.9k
3 votes
1 answer
3k views

Is it possible to use LLMs for regression tasks?

I want to use LLMs to predict edge weights in a graph based on attributes between two nodes. Is this even possible? If not, what would you recommend? I tried to look up uses of LLM in regression tasks,...
sharkeater123's user avatar
2 votes
2 answers
2k views

How does a LLM (transformer) pick words from its vocabulary?

I have a very rough understanding of the "attention/self attention" mechanism of transformer models and how this can be used to process a set of word vectors provided as an input/prompt to ...
MLBeginner's user avatar
3 votes
0 answers
2k views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
Joe Black's user avatar
  • 181
6 votes
2 answers
607 views

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
Peyman's user avatar
  • 574
2 votes
1 answer
236 views

How can a transformer encoder attend to future tokens?

What does attending to future tokens mean? From my understanding, the transformer model works by inputting a prompt and predicting the next word in a sequence and this process just keeps repeating ...
Spike's user avatar
  • 21
0 votes
1 answer
610 views

How does Chat GPT encode a question?

Chat GPT is based on a decoder-only Transformer so it does not have an encoder. Given that, how is a user's question passed as input to Chat GPT's decoder? In a regular encoder-decoder architecture, ...
joan's user avatar
  • 1
0 votes
1 answer
269 views

what are the applications scenarios for prefix decoder LMs

Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only ...
Singh's user avatar
  • 1
2 votes
0 answers
273 views

How are weight matrices in attention learned?

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight ...
Ege Demir's user avatar
0 votes
1 answer
149 views

For a transformer decoder, how exactly are K, Q, and V for each decoding step?

For a transformer decoder, how exactly are K, Q, and V for each decoding step? Assume my input prompt is "today is a" (good day). At t= 0 (generation step 0): K, Q, and V are the projections ...
wrek's user avatar
  • 183
0 votes
2 answers
137 views

How the Q,K,V be calculated in multi-head attention

I want to understand the transformer architecture, so I start with self attention and I understand their mechanism, but when I pass to the multi-head attention I find some difficulties like how ...
LAILA EL OUEDEGHYRY's user avatar
1 vote
1 answer
97 views

How are the parts of GPT connected?

Reading Stephen Wolfram's explanation of ChatGPT, it sounds as if first you train a very powerful "autocomplete" function that doesn't know anything specifically about chatbots, and then you ...
ak0000's user avatar
  • 205
0 votes
0 answers
114 views

How are the transformer encoder outputs handled?

According to the Attention Is All You Need paper, the transformer's encoder portion is described as The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The ...
rkuang25's user avatar