I know this might be a bit general question and concerning a rather active research field, much beyond my expertise, but I do believe there're some answers.
The use of NN parameters quantization can span from post-training static/dynamic quantization (PTQ) to quantization-aware training (QAT). Generally the target is cutting down FP-32 weights to UINT-8 whilst retaining overall accuracy; the benefits in performance are often sensible, yielding zero to negligible (depending on application) to few percentage drop in accuracy.
However, I hear these statements are true depending on which NN is being quantized (source). Some models indeed are very forgiving, whereas others are not even using more aggressive strategies.
(extracted from PyTorch source)
There're some networks that are very forgiving: you can do PTQ and the end result is as accurate as the FP value. [...] In some NN they're slightly more demanding, they're slightly less forgiving on the approximation [...]
My question are:
- Which categories of DNN are more suited for quantization?
- What could be one rationale why this would be? Does the size of the network (i.e. number of parameters) has any role, for instance?
- What is introducing such demands (this will depend on actual model indeed) in the forward path? Is it to be found in the activation/actual MVP or what?
My guess would be depending on achievable weights sparsity -- however, this in general (AFAIK) can be tuned forcing training constraints, so it wouldn't really answer the question.