I work with neural networks for real-time image processing on embedded softwares and I tested different architectures (Googlenet, Mobilenet, Resnet, custom networks...) and different hardware solutions (boards, processors, AI accelerators...). I noticed that the performance of the system, in terms of inference time, does not depend only on the processor but also on other factors.
For example, I have two boards from different manifacturers, B1 (with a cheap processor) and B2 (with a better processor), and two neural networks, N1 (very light with regular convolutions and fully connected layers) and N2 (very large, with inception modules and many layers). The inference time for N1 is better on B1, while for N2 it is better on N2. Moreover, it happens that, as the software is executed, the inference time changes over time.
So my question is: in an embedded system, what are the aspects that impact on the inference time, and how? I am interested not only in the hardware features but also in the neural network architecture (convolutional filter size, types of layers and so on).