I am wondering what order of magnitude estimates for the following are for companies Google and Facebook, as well as total globally.

  1. What is the rough amount of money spent to train neural networks?
  2. Number of GPU / CPU hours total used to train neural networks?
  3. What is the total energy usage (in kW hr) to train neural networks?
  4. What is the total carbon emissions produced from training neural networks?

Also, any relevant sources would be appreciated.

  • 1
    $\begingroup$ Hello. Welcome to AI SE. I think that this post is a bit problematic because you're asking too many questions, to which the answers may require a lot of effort to write. Unfortunately, someone has already attempted to give an answer to your questions. Please, at least next time, if you don't want to change this post anymore, ask only one question per post and provide as much context as necessary to understand the question, or why you're asking this question. This post may still be closed as too broad because of the mentioned reason. $\endgroup$
    – nbro
    Jul 6 '21 at 12:56

This is a really hard question to answer, as there's no telling just how much each company spends specifically on AI. It helps that Google (or rather Alphabet Inc) has a specific subsidiary company specialising in AI (DeepMind), but even with this Google may have it's own division that works on other AI projects.

You're questions are vague and vary massively depending on the task, but I'll do my best to answer them:

What is the rough amount of money spent to train neural networks?

This is very dependent on the neural network. Quite often the money spent on creating the network is the labour costs for employees/researchers to find an optimal model. Once this is found, the model can sometimes be trained on a consumer-grade GPU with no issue.

However, there are some very notable exceptions. Take GPT-3, in my opinion the closest thing we currently have to a general AI. I found an estimate that it would've cost about $4.6M USD to train using a Tesla V100 GPU (a powerful GPU specifically meant for tensor calculations) and over 355 years to train (assuming a single instance, which was not the case obviously).

For a broader overview, we can look at the annual report from DeepMind. In 2019, DeepMind made \$315 million USD in profits, but had \$849 million USD in expenses. It is likely DeepMind will continue to operate at a loss for the next 5-10 years, but after that with some proprietary AI, they could be making much much more.

Number of GPU / CPU hours total used to train neural networks?

As mentioned previously this massively depends on the task. In the case of GPT-3 it was 355 years on one GPU, but as for specifically how long it took, I am unsure. As an educated guess, I would say it can be trained in a maximum of a few months, given that any longer would mean OpenAI couldn't possibly have released GPT-3 so soon after GPT-2 (GPT-2 was released Feb 2019, GPT-3 was released July 2020).

Your last 2 questions can't easily be answered. It's massively dependent on hardware and training time. You can probably calculate these values on your own based on the training times I've provided above.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.