Tips for keeping the distribution of weights normal

I am working on a project where the Neural Network weights must be quantized on 8 or 16 bits for an embedded platform, thus I will lose some precision.

Since our platform does not have floating point arithmetic we need to quantize the weights. By quantizing i mean taking the max absolute value of the weights and divide it by the maximum signed number representable on 8 or 16 bits. this operation will give us a quantization factor $$(qf)$$. the final quantized weights will be integer$$(value * qf)$$.

If my weights are very sparse and have a very bad distribution, I lose more precision.

For example, to the left here is the distribution of weights for one layer, and to the right is the distribution of weights after I added to the loss function the Kurtosis and skew measures of the weights, and it improved a bit the shape of the distribution while keeping the same accuracy, even a bit higher.

Does anybody have any other suggestions? Has anyone tackled this problem before?

• Not sure if I understand your question correctly, but could you take the log of the values to compress the range? Feb 1 '19 at 9:30
• I edited my description. Not sure if it would help me because I work with integer arithmetic. And another downside is that the log is an expensive operation that would have to be applied also to the input...I think Feb 1 '19 at 11:36
• @FlorentinAlexandruIftimie not sure if i understand your question correctly. Here are a few doubts: 1.) I could not understand how exactly are you quantizing? 2.) And how are you initializing the weights? Are you using C or Assembly Language? 3.) What are you trying to achieve?
– user9947
Apr 19 '19 at 13:50
• @DuttaA 1. For a convolutional filter, we find the absolute max value of that tensor, name it Vmax, then if we want to have 8 bit precision, we divide 127 / Vmax, or if we want 16 bit precision we divide 32767 / Vmax both of these values will be named the quantization factor (qf). The resulting quantized tensor will have each value computed as v * qf. 2. The weights are initialized by Pytorch which is either He or Xavier (can't remember exactly) 3. I am trying to achieve a minimum quality loss of the quantized version vs floating point version Apr 20 '19 at 14:36
• @FlorentinAlexandruIftimie what is V in v*qf?
– user9947
Apr 20 '19 at 14:50

Quantizing of weights is used very often in implementing neural networks with the tensorflow library on mobile devices. The reason is, the such microcontrollers doesn't have a floating point unit and the developer is trying to increase the overall performance. Quantizing means, to map float values to integer values:

Integer Float
0      -5.0
128    0.0
256    5.0


The problem isn't trivial, because the float point numbers can have a certain minimum, maximum and precision. Maintaining the original distribution is also an issue.

To avoid such kind of problems, the best idea is to implement a floating point library from scratch. In the stackbased Forth language this can be realized in under 400 bytes. It was described in the paper “Willi Stricker: Forth Floating Point Word-Set without Floating Point Stack, 2012”

Then work with quantized network it's good idea to normalize block of them them with floating point scaling factor (16 bit float for example). The weight channel and input channel will have represenation S_f * w_n where S_f - scalar float and w_n - low-bit fixed or integer tensor or vector. Then calculating dot product put it into high-bit fixed/integer variable or float and recalculate scaling factor and low-bit representation for output. That is only if you can implement floating point (or high bit fixed) operations of cause. For low-bit representation you should clamp values to S_f*[-2^n:2^n] and that could be non-trivial. Good idea to use statistics of weights/inputs for it. There are dozen of papers on quantization of neural networks with different approaches, you may want to read some.