Why are some Neural Networks more forgiving on Quantization?

Question

I know this might be a bit general question and concerning a rather active research field, much beyond my expertise, but I do believe there're some answers.

The use of NN parameters quantization can span from post-training static/dynamic quantization (PTQ) to quantization-aware training (QAT). Generally the target is cutting down FP-32 weights to UINT-8 whilst retaining overall accuracy; the benefits in performance are often sensible, yielding zero to negligible (depending on application) to few percentage drop in accuracy.

However, I hear these statements are true depending on which NN is being quantized (source). Some models indeed are very forgiving, whereas others are not even using more aggressive strategies.

(extracted from PyTorch source)

There're some networks that are very forgiving: you can do PTQ and the end result is as accurate as the FP value. [...] In some NN they're slightly more demanding, they're slightly less forgiving on the approximation [...]

My question are:

• Which categories of DNN are more suited for quantization?
• What could be one rationale why this would be? Does the size of the network (i.e. number of parameters) has any role, for instance?
• What is introducing such demands (this will depend on actual model indeed) in the forward path? Is it to be found in the activation/actual MVP or what?

My guess would be depending on achievable weights sparsity -- however, this in general (AFAIK) can be tuned forcing training constraints, so it wouldn't really answer the question.

Answer 1

The rationale is that simpler tasks involve less complex decision boundaries, and sparse networks have more zero-valued weights that can be efficiently represented in lower precision. Larger networks with more parameters may face challenges during quantization, as there is more information to compress into fewer bits.

Ultimately, the quantization suitability depends on a combination of factors, and experimenting with different architectures

Answer 2

Regarding post-training quantization, how well you quantize your model depends on the range in which you model weights are.

if you have the following weights:
$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53]$,
you quantize and de-quantize them you get:
$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.28, -2.11, 0.33, -0.53]$.
Two minor errors on the 0.01 precision level.

However if you have the following weights with an outlier:
$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53, -67.0]$,
and you again quantize and de-quantize, then you get:
$[-0.00, -0.00, 0.00, -0.53, -0.53, -0.53, -2.11, 0.53, -0.53, -67.0]$.

Currently, one of the famous quantization algorithms for large language models is LLM.int8(). This algorithm exploits a very interesting phenomenon. It turns out that such kind of outliers are very systematic with language models with > 6.7B parameters, so they can be separated from the rest of the parameters. They quantize the regular values (which are around 99.9% of the model weights) and the outliers are not quantized. You can read more here, this is the blog of the lead author of the LLM.int8() paper.

Now, if you want to quantize your own model you have to check if you have such outliers, and combine and quantize the weights accordingly. What is so important about the LLM.int8() paper is that they provide an algorithm for any large language model. They claim that the irregularities that they found are present in every large language model.