One way to speed up a neural network is to prune the network and reducing number of neurons in each layer. What are the other methods to speed up inference?
These are some ways to speed up inference:
Reduction of float precision: This is done post-training. According to work in this segment, a very little accuracy is sacrificed for a huge benefit in memory usage reduction. Also it speeds up inference. float32 -> float8. Reference paper: https://arxiv.org/pdf/1502.02551
Using ReLU or similar small compute power activations: Benefits are obvious when you don't need to compute heavy exponents, like in tanh or sigmoid.
Binary Neural Architecture: This is new. This taps upon the ability to use binary valued weights and activations (1 bit) compared to 32 bit float counterparts. Estimation and learning is done through POPCNT and XNOR operations (for matrix products) and STE (straight through estimator) for backpropagation. You do have to sacrifice for a large number of neurons to learn the same features, but, the speed is on average 7x faster. Reference paper: https://pjreddie.com/media/files/papers/xnor.pdf
- Hardware standpoint: Using specialised hardware to compute matrix products, pretty standard examples are GPUs and TPUs.