Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [adam]

For questions about Adam, a gradient-based optimization algorithm widely used to train neural networks. It was proposed in the paper "Adam: A Method for Stochastic Optimization" (2014) by Diederik P. Kingma and Jimmy Ba.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

Is my C# Adam implementation correct?

I have some doubt because I incurred in different papers proposing different implementations. Also implementations on opensource projects looks different. In example there is a C++ library that ...
CoffeDeveloper's user avatar
2 votes
1 answer
72 views

Is it possible to use Mini-Batches with Adam optimization?

Is it possible/advised to use Mini-Batch like accumulation with Adam optimization? How would that works? Do I accumulate the loss function for each sample in the batch and then run Adam, or should I ...
CoffeDeveloper's user avatar
0 votes
0 answers
25 views

Longer DNN training times when using evolutionary algorithms

I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
knowledge_seeker's user avatar
0 votes
0 answers
193 views

What's wrong with our loss and PyTorch?

Given the samples $\vec{x_i} \in \mathbb{R}^d, i \in [1,..,l]$ where $l$ is the number of training samples, $d$ is the number of input features, the related target values $y_i \in \mathbb{R}$, and the ...
Filippo Portera's user avatar
0 votes
2 answers
38 views

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended? I have an DNN training from an infinite stream of samples, that most likely won't ...
dronus's user avatar
  • 101
0 votes
0 answers
60 views

Learned kernels in CNN seem just random patterns

I am training a classification neural network using Tensorflow2 (specifications below). The training goes well (good accuracy and no overfitting, apparently). During the training I monitor the learned ...
firion's user avatar
  • 269
2 votes
0 answers
529 views

Do learning rate schedulers conflict with or prevent convergence of the Adam optimiser?

An article on https://spell.ml says Because Adam manages learning rates internally, it's incompatible with most learning rate schedulers. Anything more complicated than simple learning warmup and/or ...
Jack G's user avatar
  • 21
4 votes
1 answer
210 views

How does OpenAI-ES use Adam?

I just read that OpenAI's ES uses Adam: "OpenAI’s ES is denoted as “OptimES” (since it uses Adam optimizer)"?? I verified they are correct using the link they posted, (see es_distributed/...
profPlum's user avatar
  • 424
1 vote
0 answers
90 views

Why are optimization algorithms for deep learning so simple?

From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
Moritz Groß's user avatar
1 vote
1 answer
353 views

How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?

I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
user50018's user avatar
1 vote
1 answer
137 views

Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?

I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation. In my experiments, I am trying different ...
Bert Gayus's user avatar
1 vote
1 answer
199 views

Why is Adam trapped in bad/suspicious local optima after the first few updates?

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
AmorFati's user avatar
2 votes
0 answers
356 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
pramesh's user avatar
  • 121
1 vote
0 answers
72 views

What is the equation of the learning rate decay in the Adam optimiser?

Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
calveeen's user avatar
  • 1,271
2 votes
1 answer
236 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
junior-flight's user avatar
2 votes
2 answers
2k views

Is the choice of the optimiser relevant when doing object detection?

Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with ...
neeraj's user avatar
  • 31
1 vote
0 answers
47 views

Are there optimizers that schedule their learning rate, momentum etc. autonomously?

I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
SpiderRico's user avatar
  • 1,000
9 votes
1 answer
1k views

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...
Dan D.'s user avatar
  • 1,293