Why momentum factor greater than 1 is a bad idea?
What are the mathematical conclusions?
Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.Sign up to join this community
Under that analogy then, momentum factor would be analogous to the friction coefficient, with 1 being max friction and 0 being no friction.
You should be able to see why there can't be friction beyond that range: if friction = 1 it would be identical to having no friction; if friction <= 0 then by conservation of energy gradient descent will not find a local minima; if friction > 1 then gradient descent would be moving backwards.
If you want only the answer to your question in particular, you can skip to the last part of the answer. To answer in detail momentum is a technically incorrect term, I would rather call it inertial learning.
Inertia - Inertia is the resistance of any physical object to any change in its position and state of motion.
beta will be the momentum term. If we expand the expression we get something like:
S_t are the gradients or
dels for a particular training example. Clearly this is for a 3 example training set.
Now why do we use momentum? As @Andreas Storvik Strauman has provided a link, you can easily delve into the mathematics for its usage. But to make a more intuitive sensehere are a few points to note:
As for your original question of why momentum term <1, here are a few points which most answers have missed:
beta > 1, the weightage for previous training examples will increase exponentially. (Like
1.01^1000 = 20959just after 1000 iterations). That maybe handled by increasing the
learning rateaccordingly, but not only it will require a lot of extra computation, it is almost mathematically impossible.
r >= 1common ratio never converges. It goes on going bigger and bigger. Also, if you can draw parallels with continuous functions, this is what we call a function which is not Absolutely integrable function.
All these leads to a single conclusion if
beta >= 1, there will be a large amount of oscillation and error will go on increasing exponentially (can probably be proven by rigorous mathematical analysis). Although it might wotk for
beta = 1(due to the Perceptron Convergence Theorem)
Let's talk about gradient decent!
So you're standing on a mountain side, and you want to get to the lowest part of this mountain. You have a notepad with you.
Although actual physics-momentum would be a good analogy here, I'm not gonna use it.
You're somewhere on this mountain side and you figure out which way is down*, and you jump once a couple of meters in that direction. How big one jump is would depend on how steep the hill is (the length of the gradient), and how much extra you push with your feet. The first time you decide to not really push that much with your feet. The SGD momentum, comes here; you write down in your note pad which direction you went, and how far (e.g. south, 4 meters).
Note: here the PHYSICAL momentum would represent the length of the gradient.
You repeat this for some time you come to a place where there are only ways upwards.
Does this mean you hit the bottom? Not necessarily; you might have gotten stuck in a valley, or "local minima". You really want to get out of this valley, but all directions are upwards, so which way should you jump?
You now take out your notebook and notice that you've been jumping south east the last 40 steps, and pretty far. You then reason that it is likely that you want to go south east. So you jump south east with a lot of thrust from your feet: This is the intuition on what momentum does;
If you have a clear "pattern" of which way is down, then this should also count!
Note: the momentum only depends on the previous step, but the previous step depends on the steps before that and so on. This is just an analogy.
For the maths, you just add a term that is the last gradient, times some constant.
Heading(t)=γ Heading(t-1)+η Gradient(t)
Where γ is the momentum factor and η is the learning rate.
Sebsastian Ruders blog on gradient descent is brilliant to learn more details of the maths of it.
For mathematical conclusions:
Heading(t)=γ Heading(t-1)+η Gradient(t)
γ > 1: From the "expression", you could infer that this case would generate echos. That the gradient of the previous step would contribute more than the actual gradient. For the upcoming step, this effect would get enhanced, and 10 steps down the road, you're stuck going in one direction.
γ < 1 makes it "converge" to a "terminal velocity", if you like. It would depend on the preceding steps less and less instead of more and more.
These effects are pretty clear in the equation you find at Ruders blog
If your momentum term was greater than one, then the notebook would overcome the actual gradient. After a few steps, you wouldn't even look at the hill; you'd go "I've only gone east so far, so I'll just continue east" with your jumps getting longer and longer. This is not good.
A high momentum term would lead you in the wrong direction (blow up and always go in the same direction), and/or oscillate around the global minima (making you jump too far).
Hope it helps :)
*: Strictly speaking, we're finding the "up" direction and go the opposite way. The "up" direction is the gradient.