1
$\begingroup$

I recently heard of GPT-3 and I don't understand how the attention models and transformers encoders and decoders work. I heard that GPT-3 can make a website from a description and write perfectly factual essays. How can it understand our world using algorithms and then recreate human-like content? How can it learn to understand a description and program in HTML?

$\endgroup$
1
  • 2
    $\begingroup$ It can't "understand" anything. It just matches patterns to something it encountered in the training data. $\endgroup$ May 7 at 16:00
0
$\begingroup$

GPT-3 (and the likes) don't really have any understanding of the semantics nor pragmatics involved in the language. However, they are good at constructing text content similar to the contents created by a person (when the texts and the concepts are not too "complicated").

$\endgroup$
5
  • $\begingroup$ But how do the different layers and math formulas work to create this type of intelligence? $\endgroup$ May 7 at 19:13
  • $\begingroup$ There are many questions regarding this issue on this platform, though I wouldn't call it "intelligence". Attention is all you need.... $\endgroup$ May 7 at 20:18
  • $\begingroup$ So then what is "Attention". I simply want to understand the working principle of GPT-3 $\endgroup$ May 7 at 21:10
  • $\begingroup$ @DragonflyRobotics We don't have a great idea, we know the general principle of neural networks is to make something that can in principle do the computation, and then spend hundreds of thousands of dollars (in the worst case) renting GPUs to tweak it quadrillions of times until it performs well. $\endgroup$
    – user253751
    Aug 10 at 16:31
  • $\begingroup$ @DragonflyRobotics The attention unit is, AFAIK, the major contribution of the transformers paper. If you want to know what attention is then I suggest you focus your question around that. $\endgroup$
    – user253751
    Aug 10 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.