2
$\begingroup$

To make an AI / LLM, like ChatGPT, you need two things:

  1. To create the LLM. This includes training it, etc. Very expensive from computation perspective.
  2. Run the LLM to answer user queries. For instance, I go to Chatgpt.com and ask "what is a dog", the LLM created in step 1 processes that question and spits out an answer.

Apparently, Nvidia (and other such companies) make special "AI" chips that are "good at AI". What does this really mean? In practice (i.e., in the real world), are Nvidia "AI chips" solving step 1, step 2, or both?

Can step 2 not be done with regular cpu chips, once the step 1 is completed? Or do we want special AI chips for both because the performance is so much better?

$\endgroup$

1 Answer 1

2
$\begingroup$

Special AI chips historically and presently are to address step 2 for one or both of two reasons:

  • The host CPU is not powerful enough for timely inference. This is a typical concern in very small devices such as mobile and battery powered.
  • To lower power consumption. In the case of always on AI in battery powered devices, keeping the main CPU running can mean poor battery life.

I have designed systems 5-6 years ago when running YOLO on an embedded CPU was 2 FPS. We plugged in the first gen Intel Neural Compute stick and it jumped to 15-20FPS. CPUs may have caught up to YOLOv3 but there will always be bigger models.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .