1
$\begingroup$

As described in this article, it was written that GPT-3 took 405 V100 years to train in 2020. I'm a bit confused about this definition, does that mean the process was accelerated like using a V100 GPU to train in 405 years?

$\endgroup$

1 Answer 1

4
$\begingroup$

The statement in which you mentioned that "GPT-3 took 405 V100 years to train" refers to the computational resources utilized in training the GPT-3 model. Specifically measured in terms of the equivalent time it would take if the GPT model is trained on a single Nvidia Tesla V100 GPU. To make it more understandable it does not mean that the training process took 405 years, but it indicates the computational intensity or amount of time it would have taken if it was trained on a single Nvidia Tesla V100 GPU.

Since GPT-3 is a very large model with 175 billion parameters, it requires very high resources in training. According to this article it takes around 1,024 Nvidia V100 GPUs to train the model, and it costs around $4.6M and 34 days to train the GPT-3 model.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .