What are general best practices or considerations in designing a model that is optimized for real-time inference?

  • $\begingroup$ The only difference between any normal model and "real time" is the fact that the the real time model needs to be as fast as possible. Rephrasing your question to "What are the quick/fast models/techniques for real time [USAGE]". This question is lacking the usage. Is this image recognition in real time? Is it audio detection? Voice generation? What do you want to do in real time, then maybe people can help $\endgroup$
    – Recessive
    Dec 17 '19 at 14:28
  • $\begingroup$ I wouldn't agree that it depends on usecase and more on how inference is genrally implemented on the GPU. So for example keeping convolution kernels small but having more layers may present very different bottlenecks in gpu implementations than having large convolution kernels and few layers. $\endgroup$ Dec 17 '19 at 15:34
  • $\begingroup$ That's true, but we still don't know what you are even doing. What do you want to do in real time $\endgroup$
    – Recessive
    Dec 18 '19 at 0:42
  • $\begingroup$ Many things. I want to start with some pixel classification - but I want to do this only to learn about inference performance and implementation specifics. My whole goal is to acquie general best practices to follow during modeling when inference performance (on mobile gpus in particular) is a concern $\endgroup$ Dec 18 '19 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.