One way to speed up a neural network is to prune the network and reducing number of neurons in each layer. What are the other methods to speed up inference?


These are some ways to speed up inference:

  • Reduction of float precision: This is done post-training. According to work in this segment, a very little accuracy is sacrificed for a huge benefit in memory usage reduction. Also it speeds up inference. float32 -> float8. Reference paper: https://arxiv.org/pdf/1502.02551

  • Using ReLU or similar small compute power activations: Benefits are obvious when you don't need to compute heavy exponents, like in tanh or sigmoid.

  • Binary Neural Architecture: This is new. This taps upon the ability to use binary valued weights and activations (1 bit) compared to 32 bit float counterparts. Estimation and learning is done through POPCNT and XNOR operations (for matrix products) and STE (straight through estimator) for backpropagation. You do have to sacrifice for a large number of neurons to learn the same features, but, the speed is on average 7x faster. Reference paper: https://pjreddie.com/media/files/papers/xnor.pdf


  • Hardware standpoint: Using specialised hardware to compute matrix products, pretty standard examples are GPUs and TPUs.
  • $\begingroup$ Binary NN's don't really work well for complex tasks and requires a lot of effort and fine tuning which probably outweighs it's benefits. $\endgroup$
    – user9947
    May 10 '19 at 5:22
  • $\begingroup$ Yeah, but they do seem promising $\endgroup$
    – Karan Shah
    May 10 '19 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.