Questions tagged [adam]
For questions about Adam, a gradient-based optimization algorithm widely used to train neural networks. It was proposed in the paper "Adam: A Method for Stochastic Optimization" (2014) by Diederik P. Kingma and Jimmy Ba.
16
questions
0
votes
0
answers
23
views
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
0
votes
2
answers
34
views
When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
I have an DNN training from an infinite stream of samples, that most likely won't ...
- 101
0
votes
0
answers
110
views
Optimal weight decay value in Adam
Is there any rule of thumb while assigning the weight_decay parameter in Adam optimizer?
As in, is it somehow related to (smaller or larger than) the learning rate ...
- 65
0
votes
0
answers
44
views
Learned kernels in CNN seem just random patterns
I am training a classification neural network using Tensorflow2 (specifications below). The training goes well (good accuracy and no overfitting, apparently). During the training I monitor the learned ...
- 269
2
votes
0
answers
392
views
Do learning rate schedulers conflict with or prevent convergence of the Adam optimiser?
An article on https://spell.ml says
Because Adam manages learning rates internally, it's incompatible with most learning rate schedulers. Anything more complicated than simple learning warmup and/or ...
- 21
4
votes
1
answer
193
views
How does OpenAI-ES use Adam?
I just read that OpenAI's ES uses Adam: "OpenAI’s ES is denoted as “OptimES” (since it uses Adam optimizer)"?? I verified they are correct using the link they posted, (see es_distributed/...
- 360
1
vote
0
answers
81
views
Why are optimization algorithms for deep learning so simple?
From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
- 133
1
vote
1
answer
257
views
How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?
I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
- 13
1
vote
1
answer
109
views
Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?
I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation.
In my experiments, I am trying different ...
- 555
1
vote
1
answer
152
views
Why is Adam trapped in bad/suspicious local optima after the first few updates?
In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write
To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
- 11
2
votes
0
answers
297
views
How to decide if gradients are vanishing?
I am trying to debug a convolutional neural network. I am seeing gradients close to zero.
How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
- 121
1
vote
0
answers
63
views
What is the equation of the learning rate decay in the Adam optimiser?
Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
- 1,251
2
votes
1
answer
224
views
How long should the state-dependent baseline for policy gradient methods be trained at each iteration?
How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods?
I'm using this equation to ...
2
votes
2
answers
1k
views
Is the choice of the optimiser relevant when doing object detection?
Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with ...
- 31
1
vote
0
answers
45
views
Are there optimizers that schedule their learning rate, momentum etc. autonomously?
I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
- 960
9
votes
1
answer
1k
views
What is the formula for the momentum and Adam optimisers?
In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is:
$$w\ -= r \times g$$
where $r$ is the ...
- 1,283