I was reading a paper recently about improving the learning of an ANN using weight normalization:

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

I wanted to check my understanding/interpretation of it with the experts here.

Normally the output of a neuron is equal to the sum of every incoming neuron multiplied by their respective weights and then pushed through an activation function:

output = activation ( SUM (x multiplied by w))

In the example in this paper it seems that instead of using each individual weight they are actually calculating a normalized weight to substitute by taking the square root of all summed weights (The Euclidean Norm?) - we'll call that Z - and then plugging it into the calculation as:

normalized_weight = ( scalar / z ) multiplied by w.

They never saw what they used for scalar btw...

Can anyone confirm if I am correct in my understanding and if not could they correct me. The maths goes a little over my head so any pseudocode is welcome.

  • $\begingroup$ Welcome to AI! You've clearly put some effort into this question, and that's what we most like to see. (I took the liberty of adding the ai-basics tag, since this is a useful, foundational question on an important topic.) $\endgroup$ – DukeZhou Feb 16 '18 at 20:06
  • 1
    $\begingroup$ Thank you! I will make sure I add that tag in the future. Plenty more to come from me. $\endgroup$ – Mike AI Feb 16 '18 at 20:26

Your interpretation is quite correct. I could not understand how it would speed up the convergence though. What they are doing is basically re-assigning the magnitude of the weight vector (also called norm of the weight vector).

To put things in the perspective, conventional approach to any Machine Learning cost function is to not only check the variations of the error with respect to a weight variable (gradient) but also add a normalization term which is lambda * (w0^2 + w1^2...). This has got few advantages:

  • The weights will not get exponentially high even though if you make some mistake (generally bouncing off to exponential costs due to wrong choice of learning rate).
  • Also the convergence is quicker somehow (maybe because you have now 2 ways to control how much weight should be given to a feature. Unimportant features weights are not only getting reduced by normal gradient, but also the gradient of the normalization term lambda * (w0^2 + w1^2...)).

In this paper, they have proposed to fix the magnitude of the weight vector. This is a good way, although I am not sure if it is better than feature normalization. By limiting the magnitude of weight to g they are fixing the resource available. The intuition is that, you have 24 hours and you have to distribute this time among subjects. You'll distribute it in a way such that your grade/knowledge is maximized. So this might be helping in the faster convergence.

Also, another intuition would be, when you are subtracting the gradient from an weight vector you use a learning rate alpha. This decides by how much weight-age of error you want to give which will be subsequently be subtracted from the weights. In this approach you are not only subtracting the weights but also using another learning rate g to scale the weight. I call this g a learning rate because yo can customize it which in turn customizes the value of weights which in turn affects the future reductions of weight gradient descent.

I am sure someone will post a better mathematical explanation of this stuff but this is all the intuition I could think of. I would be grateful if other intuitions and mathematical subtleties are pointed out. Hope this helps!

  • 2
    $\begingroup$ Thank you so much DuttaA for such a quick and detailed response. A quick follow up I had on your answer - let's say I back propogate this network. When I calculate my error, I am adjusting my actual weights as opposed to the magnitude. So I was wondering by using this technique are we not making the network train more slower? I probably need to sit down and code it to see but on a theoretical basis I am wondering if this is the case. $\endgroup$ – Mike AI Feb 16 '18 at 20:30
  • 1
    $\begingroup$ @MikeAI don't forget to upvote, and even formally accept, this answer if it's what you're looking for. $\endgroup$ – DukeZhou Feb 16 '18 at 20:39
  • $\begingroup$ @MikeAI actually I have very less knowledge of mathematics to comment on your question so I just gave the intuition what might be happening..also experts say backprop is notoriously difficult to understand why and how it is doing what it is doing. Maybe ths will help ai.stackexchange.com/questions/1479/… $\endgroup$ – DuttaA Feb 16 '18 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.