Questions tagged [adam]
For questions about Adam, a gradient-based optimization algorithm widely used to train neural networks. It was proposed in the paper "Adam: A Method for Stochastic Optimization" (2014) by Diederik P. Kingma and Jimmy Ba.
20 questions
0
votes
0
answers
12
views
setting learning rate to extremely low values in torch.optim.AdamW has no effect?
I am working on fine-tuning BLIP-2 on the RSICD dataset using LoRA. I am working on colab, using an A100. I am strangely finding that when I set the learning rate in the code below, it has no effect. ...
- 101
0
votes
0
answers
83
views
How to fix segmentation model producing black segmentation masks?
Apologies in advance for incorrect formatting, I've been kindly sent here after posting my question on StackOverflow.
We're training a segmentation model using Snapchat's template for custom ...
0
votes
0
answers
39
views
Is my C# Adam implementation correct?
I have some doubt because I incurred in different papers proposing different implementations. Also implementations on opensource projects looks different.
In example there is a C++ library that ...
- 301
2
votes
1
answer
322
views
Is it possible to use Mini-Batches with Adam optimization?
Is it possible/advised to use Mini-Batch like accumulation with Adam optimization?
How would that works?
Do I accumulate the loss function for each sample in the batch and then run Adam, or should I ...
- 301
0
votes
0
answers
31
views
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
0
votes
0
answers
193
views
What's wrong with our loss and PyTorch?
Given the samples $\vec{x_i} \in \mathbb{R}^d, i \in [1,..,l]$ where $l$ is the number of training samples, $d$ is the number of input features, the related target values $y_i \in \mathbb{R}$, and the ...
0
votes
2
answers
40
views
When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
I have an DNN training from an infinite stream of samples, that most likely won't ...
- 101
0
votes
0
answers
79
views
Learned kernels in CNN seem just random patterns
I am training a classification neural network using Tensorflow2 (specifications below). The training goes well (good accuracy and no overfitting, apparently). During the training I monitor the learned ...
- 269
2
votes
0
answers
635
views
Do learning rate schedulers conflict with or prevent convergence of the Adam optimiser?
An article on https://spell.ml says
Because Adam manages learning rates internally, it's incompatible with most learning rate schedulers. Anything more complicated than simple learning warmup and/or ...
- 21
4
votes
1
answer
229
views
How does OpenAI-ES use Adam?
I just read that OpenAI's ES uses Adam: "OpenAI’s ES is denoted as “OptimES” (since it uses Adam optimizer)"?? I verified they are correct using the link they posted, (see es_distributed/...
- 454
1
vote
0
answers
102
views
Why are optimization algorithms for deep learning so simple?
From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
- 133
1
vote
1
answer
407
views
How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?
I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
- 13
1
vote
1
answer
144
views
Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?
I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation.
In my experiments, I am trying different ...
- 635
1
vote
1
answer
245
views
Why is Adam trapped in bad/suspicious local optima after the first few updates?
In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write
To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
- 11
2
votes
0
answers
402
views
How to decide if gradients are vanishing?
I am trying to debug a convolutional neural network. I am seeing gradients close to zero.
How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
- 121
1
vote
0
answers
74
views
What is the equation of the learning rate decay in the Adam optimiser?
Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
- 1,291
2
votes
1
answer
243
views
How long should the state-dependent baseline for policy gradient methods be trained at each iteration?
How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods?
I'm using this equation to ...
2
votes
2
answers
2k
views
Is the choice of the optimiser relevant when doing object detection?
Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with ...
- 31
1
vote
0
answers
47
views
Are there optimizers that schedule their learning rate, momentum etc. autonomously?
I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
- 1,040
9
votes
1
answer
2k
views
What is the formula for the momentum and Adam optimisers?
In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is:
$$w\ -= r \times g$$
where $r$ is the ...
- 1,318