I know it cost around $4.3 million dollars to train, but how much computing power does it cost to run the finished program? IBM Watson chatbot AI only costs a few cents per chat message to use, OpeenAI Five seemed to run on a single gaming PC setup. So I'm wondering how much computing power does it need to run the finished ai program.
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ This is a programming/hardware issue, so it's off-topic here. Please, read ai.stackexchange.com/help/on-topic. This question is maybe more appropriate for Data Science SE. $\endgroup$– nbroCommented Aug 6, 2020 at 11:09
-
$\begingroup$ Not a direct answer to the question but this is also interesting in this context: reddit.com/r/MachineLearning/comments/i49jf8/… $\endgroup$– presentCommented Aug 9, 2020 at 17:43
Comments disabled on deleted / locked posts / reviews
|
2 Answers
$\begingroup$
$\endgroup$
I can't anwser your question on how much computing power you might need, but you'll need atleast a smallgrid to run the biggest model just looking at the memory requirments (175B parameters so 700GB of memory). The biggest gpu has 48 GB of vram
I've read that gtp-3 will come in eigth sizes, 125M to 175B parameters. So depending upon which one you run you'll need more or less computing power and memory.
(https://lambdalabs.com/blog/demystifying-gpt-3/)
For an idea of the size of the smallest, "The smallest GPT-3 model is roughly the size of BERT-Base and RoBERTa-Base."
$\begingroup$
$\endgroup$
I think it is premature to answer your question as OpenAI has not made GPT-3 available yet other than via a web-based API. For more information see OpenAI API.
From OpenAI will start selling its text-generation tech, and the first customers include Reddit, by James Vincent:
Access to the GPT-3 API is invitation-only, and pricing is undecided.
You can join the OpenAI wait list here: https://beta.openai.com/
I read somewhere that to load GPT-3 for inferencing requires 300GB if using half-precision floating point (FP16). There are no GPU cards today that even in a set of four will provide 300GB of video RAM. For example, the best I believe you can do in a single desktop box is four NVLinked Nvidia RTX 8000 cards on a single motherboard. Each card has 48GB of VRAM each. That would only provide a total of 192GB of VRAM.
