1
$\begingroup$

I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models.

Deep learning, generally, can involve training with supervised learning with labeled datasets which makes sense. But I also encounter references to unsupervised learning: specifically, predicting the next word in a sequence. The concept of unsupervised training with next-word prediction is unclear to me. To me, it feels like GPT is trained with supervised learning with a labeled dataset: is this correct?

$\endgroup$

2 Answers 2

2
$\begingroup$

These models are trained with unsupervised learning, but it's probably more intuitive to describe it as self-supervised learning.

Your confusion makes sense: the next-token prediction objective, which trains models to predict the next token given the previous token, is basically just doing supervised learning with the inputs being the prefix of the text and the labels being the next token. But, here, the labels are created by us, from the same corpus that the input is from! So the model is supervised by the data itself: "self-supervised" learning.

Also, to be clear, self-supervised learning is unsupervised*, you don't really have a separate set of labels---we're making them ourselves.

* I realize that these definitions can be controversial/ambiguous. I'm basing my usage on these: 1, 2, 3, 4. You could alternatively say that self-supervised learning is a separate category altogether (e.g., these slides). But, all things considered, you probably don't need to worry too much about definitions as long as you understand the underlying concepts---people will know what you're talking about either way.

$\endgroup$
2
$\begingroup$

since these labels are derived from the input data itself the pre-training phase is considered self-supervised learning, which is a subset of supervised learning, as the name would suggest. Calling this process unsupervised is an error, but many people still call it unsupervised. Terminology in every field is commonly misused because people make mistakes and have different levels of understanding of the concepts. For example people often misuse link function and the inverse link across different scientific papers.

so you are correct, self-supervised learning is a form of supervised learning because we have a target value; the one thing that defines supervised learning

edit: ok according to IBM self-supervised is a form of unsupervised learning. So maybe the answer isn't so clear. my thinking keeps coming back to autoregressive models, e.g. like ARIMA, which are considered supervised.

image the following scenario. you have stock prices Y and you use previous Y data to predict the next time step. If you did this using ARIMA I think most people would define this as a supervised task. the target value represents a perfectly accurate ground truth (e.g. the stock market price is known for a fact without any doubt). This is the case despite the target not being defined using explicit external labeling.

now if we use an LSTM or GPT, we call this self-supervised, and suddenly its an unsupervised model? the only thing we changed is the model type which shouldn't factor in to if a task is supervised/unsupervised.

the below link clarifies this more and also ellaborates the confusion around this term.

source

Though self-supervised learning is a technically a subset of unsupervised learning (as it doesn’t require labeled datasets), it’s closely related to supervised learning in that it optimizes performance against a ground truth.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .