New answers tagged

0 votes

Why do Transformer decoders use masked self attention when producing new tokens?

Its all about speed. During training: you use teacher forcing and you feed the entire target sequence $Y$ to the decoder (say of length $N$). You want the decoder to attend to ...
pi-tau's user avatar
  • 692
0 votes

What technique is used for training Large Language Models like GPT?

since these labels are derived from the input data itself the pre-training phase is considered self-supervised learning, which is a subset of supervised learning, as the name would suggest. Calling ...
Phillip Maire's user avatar
0 votes

What technique is used for training Large Language Models like GPT?

These models are trained with unsupervised learning, but it's probably more intuitive to describe it as self-supervised learning. Your confusion makes sense: the next-token prediction objective, which ...
Alexander Wan's user avatar
1 vote
Accepted

What language model can convert normal text to JSON data

I've tried Mistral-7B-Instruct-v0.2, and it worked out of the box for your example, as did Qwen1.5-1.5b. However, Qwen1.5-0.5b had issues, but I think with enough training, you can achieve decent ...
Kirill Fedyanin's user avatar
0 votes

Why does ChatGPT fail in playing "20 questions"?

This is something I studied quite extensively in this blog post: https://evanthebouncy.medium.com/llm-self-play-on-20-questions-dee7a8c63377 the main take away is LLM do not (yet) have the capacity to ...
Evan Pu's user avatar
  • 101

Top 50 recent answers are included