Is there a difference between 'inference step time' and 'inference time'?

Question

I was looking at the table presented in Keras Applications and there is a column that says Time (ms) per inference step (CPU) and another for GPU. I did a quick research on the internet and I came across some explanations of what inference time is as we can see 1-here and also 2-here.

1-The inference time is how long is takes for a forward propagation. To get the number of Frames per Second, we divide 1/inference time.


2-In deep learning, inference time is the amount of time it takes for a machine learning model to process new data and make a prediction.

But the explanations are talking about inference time and not Time per inference step, so I was wondering if there is some difference to it? Is this time related to steps_per_epoch or validation_steps from fit function?

They also mention these two lines in the Keras website :

Time per inference step is the average of 30 batches and 10 repetitions.
Batch size: 32

Does this mean that the Time per inference step is the average time it takes the model to make predictions for each batch of 32 samples as opposed to just one sample?

Thanks in advance!

Answer 1

Yes, your interpretation is correct, it's also stressed out in the documentation: "Time per inference step is the average of 30 batches and 10 repetitions.".

Be aware that the relation between batch size and inference time is not linear, so you can't halve of double the time reported on that table to estimate inference time for different batch size. And it's also very specific to the GPU used, so variations with different hardware must be expected.

Also, the table in keras documentation refers to the first definition of inference time you linked, which is more precise but less useful for real use cases. So if timing is crucial for you be sure to time also every preprocessing and post processing step that you're applying to your images and predictions.