I know it cost around $4.3 million dollars to train, but how much computing power does it cost to run the finished program? IBM Watson chatbot AI only costs a few cents per chat message to use, OpeenAI Five seemed to run on a single gaming PC setup. So I'm wondering how much computing power does it need to run the finished ai program.
1$\begingroup$ This is a programming/hardware issue, so it's off-topic here. Please, read ai.stackexchange.com/help/on-topic. This question is maybe more appropriate for Data Science SE. $\endgroup$– nbroAug 6, 2020 at 11:09
$\begingroup$ Not a direct answer to the question but this is also interesting in this context: reddit.com/r/MachineLearning/comments/i49jf8/… $\endgroup$– presentAug 9, 2020 at 17:43
I can't anwser your question on how much computing power you might need, but you'll need atleast a smallgrid to run the biggest model just looking at the memory requirments (175B parameters so 700GB of memory). The biggest gpu has 48 GB of vram
I've read that gtp-3 will come in eigth sizes, 125M to 175B parameters. So depending upon which one you run you'll need more or less computing power and memory.
For an idea of the size of the smallest, "The smallest GPT-3 model is roughly the size of BERT-Base and RoBERTa-Base."
I think it is premature to answer your question as OpenAI has not made GPT-3 available yet other than via a web-based API. For more information see OpenAI API.
From OpenAI will start selling its text-generation tech, and the first customers include Reddit, by James Vincent:
Access to the GPT-3 API is invitation-only, and pricing is undecided.
You can join the OpenAI wait list here: https://beta.openai.com/
I read somewhere that to load GPT-3 for inferencing requires 300GB if using half-precision floating point (FP16). There are no GPU cards today that even in a set of four will provide 300GB of video RAM. For example, the best I believe you can do in a single desktop box is four NVLinked Nvidia RTX 8000 cards on a single motherboard. Each card has 48GB of VRAM each. That would only provide a total of 192GB of VRAM.