Questions tagged [adam]

For questions about Adam, a gradient-based optimization algorithm widely used to train neural networks. It was proposed in the paper "Adam: A Method for Stochastic Optimization" (2014) by Diederik P. Kingma and Jimmy Ba.

Filter by
Sorted by
Tagged with
1
vote
0answers
27 views

What is the equation of the learning rate decay in the Adam optimiser?

Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
2
votes
1answer
108 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
2
votes
2answers
54 views

Is the choice of the optimiser relevant when doing object detection?

Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with ...
1
vote
0answers
35 views

Are there optimizers that schedule their learning rate, momentum etc. autonomously?

I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
6
votes
1answer
625 views

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...