To me it looks like GPT-4 is based on GPT-3.
On the other hand, there were rumors that training of GPT-3 was done with errors, but re-train was impossible due to the costs.
$\begingroup$
$\endgroup$
3
-
$\begingroup$ are you thinking of the "glitch tokens"? Those can be fixed without re-training. $\endgroup$– user253751Mar 16 at 22:39
-
$\begingroup$ @user253751 what is this? $\endgroup$– AnixxMar 16 at 23:43
-
2$\begingroup$ Glitch tokens - interesting stuff! If this is the bug you were talking about, it doesn't require re-training. $\endgroup$– user253751Mar 17 at 0:14
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
GPT-4 is a transformer like GPT-3 and any other GPT. The training is certainly new, because it has a different size, you just can not transfer GPT-3 weights into an GPT-4 to continue training.
The details of the implementation are currently not known, the published report about it is not a scientific paper about how GPT-4 works, it basically states "we do not say how it works", see below.
The size of various components certainly differ, at least the number of weights in one part, but it could also be larger everywhere. The prompt length (context size) and the maximal output size increased a lot, from 4000 tokens to 8000 or even 32000 tokens. I would expect there are some other minor differences.
Other major differences can be the amount of training data, and the compute used for training. My personal speculation is that GPT-4 used a lot more compute for training, with possibly a similar amount of training data as GPT-3.
From the "GPT-4 Technical Report", section 2:
Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.
-
1$\begingroup$ Saying that GPT-3 is just a bigger GPT and GPT-2 is correct, but saying GPT-4 is the same as GPT-3, conceptually, I believe is incorrect. You cannot feed an image like a text without processing steps, such as like ViT's way. That makes GPT-4 differ from GPT-3, albeit small. $\endgroup$ Mar 22 at 8:38
-
$\begingroup$ Yes, I added discussion of the differences. $\endgroup$ Mar 22 at 14:07
$\begingroup$
$\endgroup$
7
GPT-4 is largely based on GPT-3. From the GPT-4 Technical Report:
GPT-4 is a Transformer-style model [39]
The transformer-style model originates from the paper Attention Is All You Need, which lays the foundation of GPT, GPT-2, and GPT-3.
However, there is one significant change: GPT-4 accepts images as inputs. This has been observed in the paper Learning Transferable Visual Models From Natural Language Supervision by OpenAI. We can safely predict that GPT-4 is based on GPT-3 + CLIP.
I say predict because the authors decided not to publish the architecture or anything:
Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.
However, we have no details regarding whether it is trained from scratch, or use any technique.
-
-
$\begingroup$
this report contains no further details about [...] training method$\endgroup$ Mar 19 at 11:35 -
$\begingroup$ @Anixx As I already said in the removed answer, GPT-4 is just a retrained GPT-3 with more fine tuning. It's a matter of what has changed in the training data set. For example, it seem that the previously mentioned
glitch tokeshave now been removed, which is likely an effect of removing dumb scraping data, such as from the counting sub-reddit, or from scraping random debug log files, etc. In other words, the training is done from scratch, but it's likeley to be done on mostly the same data, adjusted to remove problem points, as discovered by humans from previous version. $\endgroup$ Mar 19 at 12:42 -
$\begingroup$ @not2qubit glitch tokens are an effect of having tokens in the token set which were not, or rarely, in the training data, as they result from spam data which was removed before training, but after selecting the tokens. They can be removed from the token splitter which converts text into tokens without needing to re-train. $\endgroup$ Mar 21 at 22:00
-
$\begingroup$ @not2qubit
GPT-4 is just a retrained GPT-3 with more fine tuningis this what you think or it is something the authors say? Why does removing the glitch token infers GPT-4 is just a fine-tune GPT-3? $\endgroup$ Mar 22 at 4:13