What are the aspects that most impact on the inference time for neural networks in embedded systems?

Question

I work with neural networks for real-time image processing on embedded softwares and I tested different architectures (Googlenet, Mobilenet, Resnet, custom networks...) and different hardware solutions (boards, processors, AI accelerators...). I noticed that the performance of the system, in terms of inference time, does not depend only on the processor but also on other factors.

For example, I have two boards from different manifacturers, B1 (with a cheap processor) and B2 (with a better processor), and two neural networks, N1 (very light with regular convolutions and fully connected layers) and N2 (very large, with inception modules and many layers). The inference time for N1 is better on B1, while for N2 it is better on N2. Moreover, it happens that, as the software is executed, the inference time changes over time.

So my question is: in an embedded system, what are the aspects that impact on the inference time, and how? I am interested not only in the hardware features but also in the neural network architecture (convolutional filter size, types of layers and so on).

Answer 1

You can expect that the inference time will strongly depend on particular hardware and software present on your platform. First, GPU equipped devices (eg NVidia TX) will outperform non-GPU equipped devices (eg. Intel Movidius). Second, software support (eg. cudnn, TensorRT) will make dramatic further impact.

For instance, we have measured the inference time of two convolutional models. The model A requires 250% more floating point operations than the model B. Yet, the two models take roughly the same time to evaluate on our device, since the layers of model A are better optimized in software. Conclusion: algorithmic complexity and practical execution time on a particular computing platform are not bound to be proportional any more.