Questions tagged [adam]

For questions about Adam, a gradient-based optimization algorithm widely used to train neural networks. It was proposed in the paper "Adam: A Method for Stochastic Optimization" (2014) by Diederik P. Kingma and Jimmy Ba.

Filter by
Sorted by
Tagged with
0 votes
0 answers

Longer DNN training times when using evolutionary algorithms

I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
knowledge_seeker's user avatar
0 votes
2 answers

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended? I have an DNN training from an infinite stream of samples, that most likely won't ...
dronus's user avatar
  • 101
0 votes
0 answers

Optimal weight decay value in Adam

Is there any rule of thumb while assigning the weight_decay parameter in Adam optimizer? As in, is it somehow related to (smaller or larger than) the learning rate ...
helloworld's user avatar
0 votes
0 answers

Learned kernels in CNN seem just random patterns

I am training a classification neural network using Tensorflow2 (specifications below). The training goes well (good accuracy and no overfitting, apparently). During the training I monitor the learned ...
firion's user avatar
  • 269
2 votes
0 answers

Do learning rate schedulers conflict with or prevent convergence of the Adam optimiser?

An article on says Because Adam manages learning rates internally, it's incompatible with most learning rate schedulers. Anything more complicated than simple learning warmup and/or ...
Jack G's user avatar
  • 21
4 votes
1 answer

How does OpenAI-ES use Adam?

I just read that OpenAI's ES uses Adam: "OpenAI’s ES is denoted as “OptimES” (since it uses Adam optimizer)"?? I verified they are correct using the link they posted, (see es_distributed/...
profPlum's user avatar
  • 360
1 vote
0 answers

Why are optimization algorithms for deep learning so simple?

From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
Moritz Groß's user avatar
1 vote
1 answer

How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?

I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
user50018's user avatar
1 vote
1 answer

Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?

I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation. In my experiments, I am trying different ...
Bert Gayus's user avatar
1 vote
1 answer

Why is Adam trapped in bad/suspicious local optima after the first few updates?

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
AmorFati's user avatar
2 votes
0 answers

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
pramesh's user avatar
  • 121
1 vote
0 answers

What is the equation of the learning rate decay in the Adam optimiser?

Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
calveeen's user avatar
  • 1,251
2 votes
1 answer

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
junior-flight's user avatar
2 votes
2 answers

Is the choice of the optimiser relevant when doing object detection?

Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with ...
neeraj's user avatar
  • 31
1 vote
0 answers

Are there optimizers that schedule their learning rate, momentum etc. autonomously?

I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
SpiderRico's user avatar
9 votes
1 answer

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...
Dee's user avatar
  • 1,283