8
$\begingroup$

To me it looks like GPT-4 is based on GPT-3.

On the other hand, there were rumors that training of GPT-3 was done with errors, but re-train was impossible due to the costs.

$\endgroup$
3
  • $\begingroup$ are you thinking of the "glitch tokens"? Those can be fixed without re-training. $\endgroup$ Commented Mar 16, 2023 at 22:39
  • $\begingroup$ @user253751 what is this? $\endgroup$
    – Anixx
    Commented Mar 16, 2023 at 23:43
  • 2
    $\begingroup$ Glitch tokens - interesting stuff! If this is the bug you were talking about, it doesn't require re-training. $\endgroup$ Commented Mar 17, 2023 at 0:14

2 Answers 2

4
$\begingroup$

GPT-4 is a transformer like GPT-3 and any other GPT. The training is certainly new, because it has a different size, you just can not transfer GPT-3 weights into an GPT-4 to continue training.

The details of the implementation are currently not known, the published report about it is not a scientific paper about how GPT-4 works, it basically states "we do not say how it works", see below.

The size of various components certainly differ, at least the number of weights in one part, but it could also be larger everywhere. The prompt length (context size) and the maximal output size increased a lot, from 4000 tokens to 8000 or even 32000 tokens. I would expect there are some other minor differences.

Other major differences can be the amount of training data, and the compute used for training. My personal speculation is that GPT-4 used a lot more compute for training, with possibly a similar amount of training data as GPT-3.


From the "GPT-4 Technical Report", section 2:

Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

$\endgroup$
4
  • 1
    $\begingroup$ Saying that GPT-3 is just a bigger GPT and GPT-2 is correct, but saying GPT-4 is the same as GPT-3, conceptually, I believe is incorrect. You cannot feed an image like a text without processing steps, such as like ViT's way. That makes GPT-4 differ from GPT-3, albeit small. $\endgroup$ Commented Mar 22, 2023 at 8:38
  • $\begingroup$ Yes, I added discussion of the differences. $\endgroup$ Commented Mar 22, 2023 at 14:07
  • $\begingroup$ "you just can not transfer GPT-3 weights into an GPT-4 to continue training" - actually, you could, as long as what you transfer is compatible in mathematical terms (i.e. you can still multiply the matrices/tensors). In fact, a form of transfer learning is based on the idea of freezing some weights then connecting a new compatible set of weights. It's very well possible that they are doing something like that or some form of model compression, as training from scratch may take a long time. $\endgroup$
    – nbro
    Commented May 4, 2023 at 20:34
  • $\begingroup$ @nbro If GPT-4 would be constructed to be kind of an extension of GPT-3, that would be possible, and very interesting. But that would require very different CNNs, I think you can not just hammer a small transformer into a larger one. We do not know what they actually did, but trying to build GPT-4 based of a learned model of a smaller CNN sounds like big scientific progress we would know of. Hmm... But then, what would happen when you just take the weights you have from an old smaller model, and use the as initialization as the beginning, together with 20 times the amount of noise or zeros? $\endgroup$ Commented May 4, 2023 at 23:14
6
$\begingroup$

GPT-4 is largely based on GPT-3. From the GPT-4 Technical Report:

GPT-4 is a Transformer-style model [39]

The transformer-style model originates from the paper Attention Is All You Need, which lays the foundation of GPT, GPT-2, and GPT-3.

However, there is one significant change: GPT-4 accepts images as inputs. This has been observed in the paper Learning Transferable Visual Models From Natural Language Supervision by OpenAI. We can safely predict that GPT-4 is based on GPT-3 + CLIP.

I say predict because the authors decided not to publish the architecture or anything:

Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

However, we have no details regarding whether it is trained from scratch, or use any technique.

$\endgroup$
7
  • $\begingroup$ But did they re-train from scratch it or not? $\endgroup$
    – Anixx
    Commented Mar 19, 2023 at 11:14
  • $\begingroup$ this report contains no further details about [...] training method $\endgroup$ Commented Mar 19, 2023 at 11:35
  • $\begingroup$ @Anixx As I already said in the removed answer, GPT-4 is just a retrained GPT-3 with more fine tuning. It's a matter of what has changed in the training data set. For example, it seem that the previously mentioned glitch tokes have now been removed, which is likely an effect of removing dumb scraping data, such as from the counting sub-reddit, or from scraping random debug log files, etc. In other words, the training is done from scratch, but it's likeley to be done on mostly the same data, adjusted to remove problem points, as discovered by humans from previous version. $\endgroup$
    – not2qubit
    Commented Mar 19, 2023 at 12:42
  • $\begingroup$ @not2qubit glitch tokens are an effect of having tokens in the token set which were not, or rarely, in the training data, as they result from spam data which was removed before training, but after selecting the tokens. They can be removed from the token splitter which converts text into tokens without needing to re-train. $\endgroup$ Commented Mar 21, 2023 at 22:00
  • $\begingroup$ @not2qubit GPT-4 is just a retrained GPT-3 with more fine tuning is this what you think or it is something the authors say? Why does removing the glitch token infers GPT-4 is just a fine-tune GPT-3? $\endgroup$ Commented Mar 22, 2023 at 4:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .