Consider that a model using FP16 precision is quantized to a lower precision like INT8. Does this reduce the accuracy of the model? From what I know it is designed to reduce the size and required RAM to run the models.


1 Answer 1


Quantization is an active research area, so there's going to be a number of ways to do quantization.

Huggingface has a nice blogpost with an overview of 8 bit quantization. I'll give a rough summary of some important points here.  


With FP16 -> INT8 quantization specifically, you want to project all FP16 numbers into the 256 possible INT8 values. You can't directly round your parameters into integers (otherwise you'll just get a bunch of zeros!) but you can rescale your floating point values to take full advantage of the 256 integers you're limited to.

As a concrete example,

...if my range is -1.0…1.0 and I want to quantize into the range -127…127, I want to scale by the factor of 127 and then round it into the 8-bit precision. To retrieve the original value, you would need to divide the int8 value by that same quantization factor of 127. For example, the value 0.3 would be scaled to 0.3*127 = 38.1. Through rounding, we get the value of 38. If we reverse this, we get 38/127=0.2992 – we have a quantization error of 0.008 in this example.

"Quantization error" here refers to the difference between the original value and the quantized representation. With this quantization method, limited to only 256 possible quantized values, the closest value we can map 0.3 to is 0.2992.

Quantization will pretty much always result in some amount of quantization error---by lowering the precision you're artificially adding noise to your weights in the form of quantization errors. The interesting question is in knowing what kinds of errors models are robust to. We can then take advantage of this to devise quantization methods that only add this kind of noise that doesn't affect model performance.

As a simple example, consider again the previous quantization method, quantizing floating point numbers to integers from -127 to 127 (by multiplying by 127). If, it turns out that your model's parameters are always between 0 and 1, then this method of quantization would be extremely suboptimal: there's no use dedicating 127 integers to representing negative numbers that you'll never see in practice.

The key takeaway here is that the quality of your quantization depends on taking advantage of knowledge about the range of values that you're quantizing. By quantizing, you're sacrificing precision for values that you know you'll never see in favor of accurately representing values that you will see.


The post, and accompanying paper discuss two methods to do this: absmax and zeropoint quantization. Absmax is the more popular method. It just rescales based on the absolute value of the maximum of the values you're quantizing. For example, if you have the values 0.4, -0.3, -1.2, 0.5. You'd first divide by the max absolute value: 0.4/1.2, -0.3/1.2, -1.2/1.2, 0.5/1.2, giving you values between -1 and 1. You can then rescale to match the range of INT8 by simplying multiplying by 127. This method tends to be worse for asymmetric distributions (e.g., from 0 to 1 as opposed to -1 to 1) as negative integers won't be used. Zeropoint quantization better handles these values.


This can work, but there a couple other tricks you need if you want quantization to work consistent, especially across larger models. Again, these methods were originally proposed in the accompanying paper.

If you have a large set of values, you'd be able to quantize more accurately if you were to quantize smaller subsets of your data separately versus quantizing the entire set of data in one go. This is especially important if you have subsets that contain different ranges of values. Here, the authors do this by independently quantizing the rows and columns of matrices being multiplied.

Another trick is that large transformer models tend to have outlier values that are tough to quantize. e.g., if you have a bunch of values between -1 and 1 and single value = 1000, you either need to sacrifice precision significantly by rescaling everything by 1000 or you'll need to get rid of the outlier value. The authors instead propose using fp16 precision for outlier values and int8 for everything else.


The table below shows the results from the paper (for perplexity; lower is better): Quantization and perplexity results for the LLM.int8() paper by Dettmers et al., 2022.

With the right quantization method, you can retain most of the accuracy of LLMs despite introducing quantization errors.


You must log in to answer this question.