8
$\begingroup$

It (Adagrad) adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.

From Sebastian Ruder's Blog

If a parameter is associated with an infrequent feature then yes, it is more important to focus on properly adjusting that parameter since it is more decisive in classification problems. But how does making the learning rate higher in this situation help?

If it only changes the size of the movement in the dimension of the parameter (makes it larger) wouldn't that make things even more imprecise? Since the network depends more on those infrequent features, shouldn't adjusting those parameters be done more precisely instead of just faster? The more decisive parameters should have a higher "slope", thus why should they also have high learning rates? I must be missing something, but what is it?

Further, in the article, the formula for parameter adjustments with Adagrad is given. Where exactly in that formula do you find the information about the frequency of a parameter? There must be a relationship between the gradients of a parameter and the frequency of features associated with it because it's the gradients that play an important role in the formula. What is that relationship?

TLDR: I don't understand both the purpose and formula behind Adagrad. What is an intuitive explanation of it that also provides an answer to the questions above, or shows why they are irrelevant?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I'm fairly sure lecture 3 of cs231n on youtube provides a good explanation at some point. I'm just not sure if this is the specific lecture or not. $\endgroup$
    – Recessive
    Aug 19, 2019 at 3:17

1 Answer 1

Reset to default
1
$\begingroup$

I found a somewhat more accessible introduction here:

https://medium.com/konvergen/an-introduction-to-adagrad-f130ae871827

Let me start from the last part of your question. The frequency of a parameter is in G_t, which is the accumulated sum of squared gradients from all the time steps up to step t. If the gradient vanishes in many of the previous steps, then you divide the learning rate with a smaller number for that parameter.

And for the first part, you want the parameter that is more frequent to have a smaller learning rate as it is updated on more iterations compared to a parameter which is updated only a small number of times.

Improve this answer
$\endgroup$
2
  • $\begingroup$ For the first part: Maybe its just the fact that SGD doesnt update the neurons associated with the infrequent features much because they provide relatively small improvement (low gradient) and thus that must be compensated for with a higher learning rate? So basically, how does the frequency of feature influence frequency of adjustment (and not only the gradient and thus adjustment itself) if adjustments don't occur after every example, but after a mini-batch of examples where an example with the infrequent feature (for the sake of simplicity) has occured? $\endgroup$
    – DaddyMike
    Aug 20, 2019 at 12:08
  • 1
    $\begingroup$ A good point for which I have no answer. I could not find anything about this in the original paper by Duchi et al, or the deep learning book of Goodfellow. I suppose the parameters have to be sparse enough (compared to mini batch size) for the algorithm to work effectively. – serali 9 mins ago Delete $\endgroup$
    – serali
    Aug 20, 2019 at 13:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .