4
$\begingroup$

The recent advances in machine learning were mostly achieved by the hardware, and the hardware is said to continue driving the development of AI, but I was still shocked by this thread which reads that the most expensive and largest model would cost 1B dollars in 2025. And I learned that universities are suffering from an academic AI brain drain partly due to the scarce hardware resources.

enter image description here

Some people proposed the so-called Green AI that encourages sustainable AI development but provides few constructive methods to prevent the trend.

I wonder if the redder and redder AI would be in fact truly inevitable. It seems to me that all companies should build an expensive compute infrastructure to be competitive, but I think the investment would be very risky since most companies cannot get a higher return.

But on the other hand, we human beings have evolved tens of millions of years or billions of years(life) on earth with hundreds of billions of brains that have ever lived on earth as a whole "human brain". The biological wetware seems much much redder than the nowadays hardware and has consumed much much more energy than all the supercomputers. To make machines as intelligent as we humans shouldn't we pay as high a price? It reminds me of the NFL theorem but it should be imprecise in this scenario.

So, will there be some promising techniques on the algorithm side that can make AI greener, affordable and sustainable in the future? If not, could anyone please explain why AI should be unavoidably red and inevitably redder?

$\endgroup$
8
  • $\begingroup$ Quantum computing and Neuromorphic computing. Both are low power (not sure about QC though). $\endgroup$ – DuttaA Feb 18 at 3:20
  • $\begingroup$ @DuttaA It seems that quantum computing would be equally expensive? $\endgroup$ – Lerner Zhang Feb 18 at 3:26
  • 1
    $\begingroup$ Maybe I am not sure about QC but since it is in a nascent stage I guess energy optimization will be possible. Whereas Neuromorphic computing will hands down save huge amounts of energy. $\endgroup$ – DuttaA Feb 18 at 7:54
  • $\begingroup$ @DuttaA Glad to know that and let me learn what that is. $\endgroup$ – Lerner Zhang Feb 18 at 8:36
  • $\begingroup$ ai.stackexchange.com/questions/7328/… I wrote a bit outdated answer, but the energy savings will be mainly because memristors are passive memories and also one can learn directly from data as analog input values rather than pre processing it to digital and running NNs and what not. This is the basic idea (although this is also a nascent field and has its fair share of challenges) $\endgroup$ – DuttaA Feb 18 at 9:19
3
$\begingroup$

As far as I know, green AI and red AI are very recent terms and/or research areas, but their importance will or should definitely be more highlighted in the coming years (for obvious reasons).

I don't know how many people are actively and directly researching this topic, but, in the past, I've already read a research paper on the topic, so a few people are already trying to at least raise awareness about these issues. Moreover, they also suggest that researchers report the number of floating-point operations (FLOPs) performed during the experiments, in addition to the usual performance metrics (such as the accuracy). I don't remember all the details of this paper, but I really recommend that you read it. This another possibly useful paper that people interested in this topic might want to read.

Although I'm currently not doing research on this topic, from an algorithmic perspective, improvements in the following areas will definitely contribute to greener AI

  • zero-, one- or few-shot learning
  • transfer learning
  • sample efficiency (in reinforcement learning)

Why? Because learning from fewer samples typically means fewer computations, so fewer emissions.

So, I think that making AI greener is definitely possible, as it's possible to use an electric car rather than a petrol/diesel car, although more research on the topics mentioned above (and other topics) needs to be done to make it practically useful.

$\endgroup$
2
  • $\begingroup$ Zero-, or- or few-shot learning and transfer learning should be all based on expensive pre-training. So far as I know some pre-trained models cannot be employed for specific tasks, please refer to the Reasoning fine-tuning performance part. The pre-trained models are too large to fine-tune. $\endgroup$ – Lerner Zhang Feb 18 at 13:39
  • $\begingroup$ @LernerZhang You may be right, but if you can re-use some models for different tasks, without training them from scratch, that's already a win. Humans don't learn everything from scratch, so I think that making AI behave/learn more as humans do may actually be very beneficial. I am not an expert in the topic, but, from what I remember having read around, humans or the human brains are a lot more efficient than computers. $\endgroup$ – nbro Feb 18 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.