When to use Pruning, Quantization , Distillation and others when optimizing speed

Question

I want to understand how to optimize models for inference speed and am seeking some advice and best practices for the same.

I am a little bit aware of the concepts of pruning, quantization, and distillation. But not aware of the scenarios and conditions under which one to choose and which will be most effective. My main target is to improve inference speed without reducing the model's accuracy.

My understanding is as follows:

Pruning: It involves removing non-critical parts of the model (layer or neuron) which reduces the complexity and is helpful in over-parameterized networks. But not clear on the best scenarios or model types where pruning is most effective in terms of trade-off.

Quantization: It reduces the precision of the model's parameters which leads to faster calculation and reduces the model size as well. I know that this is the most effective way to reduce the inference time. However, I am not sure how to balance the level of quantization and its effects.

Distillation: It is mostly Used to transfer knowledge from a large or complex model to a smaller and more efficient model. I am curious to know in what cases the distillation outperforms other optimization methods and how can we maintain performance fidelity in the distilled model.

The questions I want to ask are:

Pruning: Want to know some case studies where pruning is effectively Implemented? How to decide on how much degree we do pruning?

Quantization: What are the best practices for deciding the level of quantization? Some use cases or architectures where quantization is exceptional

Distillation: In which situation or model type distillation is a better choice than Quantization and Pruning? Are there any models or use cases where distillation is helpful?

Also, I want to know about other optimization techniques or strategies which I missed and are helpful for model inference speed optimization.

Please add any techniques which I missed over here.

Answer 1

There are many survey papers on the topic of efficient deep learning. In general the decisions for pruning and quantisation are very dependant on your target hardware and downstream task.

For example:

1. Structured pruning (filter/row-wise) is much more suited for GPUs that are optimised for dense matrix multiplication. This achieves a much lower level of sparsity than unstructured pruning. Another point worth mentioning is that pruning might not even yield improved any latency reduction when you are not memory bound.

2. Very low-bit widths are ill-suited for GPUs and CPUs, but can be implemented on FPGAS. Training very low bit-widths requires rethinking the training and architecture significantly. Generally, you don't lose much going to 8/16 bits.

3. Distillation allows you to design your student architecture with your target hardware/setting in mind but its efficacy is very task-specific. For example, distillation is generally more effective on discriminative tasks such as classification. Many works have distilled from full-precision to quantised student models, which shows that you don't need to choose one or the other.

Finally, there are lots of other works such as early exiting, dynamic inference, and low-rank decomposition that can be used.