-1
$\begingroup$

I've watched Tesla AI Day 2021 and there was a question Tesla staff tried to answer, but I did not quite understand the question (Note: quote taken from autogenerated subtitles, I do not hear differently, but may you will):

or you'd be training a lot more complex models which would be potentially significantly more expensive to run at inference time on the cars

I've found a definition of "inference time" in How to Optimize a Deep Learning Model for faster Inference?

The inference time is how long is takes for a forward propagation

But what does "AT inference time on the cars" mean? Is it just badly worded, or does this "at" actually add proper meaning? Also, does it make sense to run training models on the cars themselves and what can that phrase mean? Overall I do not make sense of the question. Do you?

Note: I'm not a native English speaker.

$\endgroup$
0
2
$\begingroup$

"At inference time" means "when you perform inference". If "inference" is a synonym for "forward pass" (aka "forward propagation") (which is not always the case in ML), then "at inference time", again, means "when you perform the forward pass". "At" is just a preposition in English and it's often associated with location or time.

So, the sentence

you'd be training a lot more complex models which would be potentially significantly more expensive to run at inference time on the cars

can be rewritten as follows

you'd be training a lot more complex models which would be potentially significantly more expensive to run when performing the forward pass (i.e computing the predictions, for example, whether the traffic light is green or red) on the cars

$\endgroup$
8
  • $\begingroup$ thank you. my confusion was due to at meaning more "at point of time" whereas "forward propagation" means some duration. What your answer tells me is that, in the context of training, duration of one "forward propagation" is small enough to be talked about as "at point of time". Am I correct at that understanding of the answer? $\endgroup$ Dec 11 '21 at 21:27
  • $\begingroup$ The "forward propagation" is just the algorithm that we use to compute the predictions or output of the neural network. "At inference time" means when we run this algorithm (i.e. not when we're training the neural network). $\endgroup$
    – nbro
    Dec 11 '21 at 21:39
  • $\begingroup$ "(i.e. not when we're training " that comment re-ignited my confusion with the question in the video. second reason (not clearly stated in the question, I see that) is the person compares "training models" (on supercomputer, as far as I recall) with doing it "at inference time on the cars". You write second part on the cars is not training. Why does he compare costs in that case? $\endgroup$ Dec 11 '21 at 21:48
  • $\begingroup$ @Martian2020 "at inference time on the cars" means when you're using the trained models to perform predictions about the traffic situation or whatever. $\endgroup$
    – nbro
    Dec 11 '21 at 22:17
  • 1
    $\begingroup$ Ok. I see now where your other doubts lie. They are talking about training multiple models and then they say that this would imply that it's more expensive to run "at inference time". The only explanation that I have is that having multiple models not only makes training more expensive but also making predictions (i.e. inference) more expensive (once the models have been trained, because we need to perform multiple forward passes: one for each model?). This is just a guess of what they could have meant. I didn't watch that video. $\endgroup$
    – nbro
    Dec 11 '21 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.