Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, each weight will get updated differently based on the accumulated squared gradients in their respective dimensions, even though $\alpha$ might be constant. There are other StackOverflow posts that have said that Adam has a built-in learning rate decay. In the original paper also, the authors of adam paper says that the learning rate at time step $t$ decays based on the equation $$\alpha_t = \alpha \cdot \frac{\sqrt{1-\beta_2^t}}{{1-\beta_1^t}}$$

Is the second equation the learning rate decay that has been built into the Adam algorithm?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.