Tips for keeping the distribution of weights normal

I am working on a project where the Neural Network weights must be quantized on 8 or 16 bits for an embedded platform, thus I will lose some precision.

Since our platform does not have floating point arithmetic we need to quantize the weights. By quantizing i mean taking the max absolute value of the weights and divide it by the maximum signed number representable on 8 or 16 bits. this operation will give us a quantization factor $$(qf)$$. the final quantized weights will be integer$$(value * qf)$$.

If my weights are very sparse and have a very bad distribution, I lose more precision.

For example, to the left here is the distribution of weights for one layer, and to the right is the distribution of weights after I added to the loss function the Kurtosis and skew measures of the weights, and it improved a bit the shape of the distribution while keeping the same accuracy, even a bit higher. Does anybody have any other suggestions? Has anyone tackled this problem before?

• Not sure if I understand your question correctly, but could you take the log of the values to compress the range? Feb 1 '19 at 9:30
• I edited my description. Not sure if it would help me because I work with integer arithmetic. And another downside is that the log is an expensive operation that would have to be applied also to the input...I think Feb 1 '19 at 11:36
• @FlorentinAlexandruIftimie not sure if i understand your question correctly. Here are a few doubts: 1.) I could not understand how exactly are you quantizing? 2.) And how are you initializing the weights? Are you using C or Assembly Language? 3.) What are you trying to achieve?
– user9947
Apr 19 '19 at 13:50
• @DuttaA 1. For a convolutional filter, we find the absolute max value of that tensor, name it Vmax, then if we want to have 8 bit precision, we divide 127 / Vmax, or if we want 16 bit precision we divide 32767 / Vmax both of these values will be named the quantization factor (qf). The resulting quantized tensor will have each value computed as v * qf. 2. The weights are initialized by Pytorch which is either He or Xavier (can't remember exactly) 3. I am trying to achieve a minimum quality loss of the quantized version vs floating point version Apr 20 '19 at 14:36
• @FlorentinAlexandruIftimie what is V in v*qf?
– user9947
Apr 20 '19 at 14:50