Questions tagged [adam]

For questions about Adam, a gradient-based optimization algorithm widely used to train neural networks. It was proposed in the paper "Adam: A Method for Stochastic Optimization" (2014) by Diederik P. Kingma and Jimmy Ba.

Filter by
Sorted by
Tagged with
1
vote
0answers
32 views

Why are optimization algorithms for deep learning so simple?

From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
1
vote
1answer
21 views

How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?

I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
0
votes
0answers
25 views

Does Adam remember gradient momentum from previous fits, or this only works for batches?

I am training my model in loop using tensorflow and keras, iterating over data etc. Simple example ...
0
votes
0answers
22 views

How is the marginal likelihood of wide valleys higher than that of narrow valleys when optimizing a cost function?

I am reading the paper Entropy-sgd: Biasing gradient descent into wide valleys by Chaudhari et al. From what I understand, wide valleys tend to generalize better than sharp ones because they are ...
1
vote
1answer
48 views

Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?

I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation. In my experiments, I am trying different ...
1
vote
1answer
44 views

Why is Adam trapped in bad/suspicious local optima after the first few updates?

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
2
votes
0answers
73 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
1
vote
0answers
30 views

What is the equation of the learning rate decay in the Adam optimiser?

Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
2
votes
1answer
158 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
2
votes
2answers
317 views

Is the choice of the optimiser relevant when doing object detection?

Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with ...
1
vote
0answers
37 views

Are there optimizers that schedule their learning rate, momentum etc. autonomously?

I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
8
votes
1answer
781 views

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...