9
votes
Accepted
What is the formula for the momentum and Adam optimisers?
I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. ...
- 33.3k
3
votes
Accepted
Is it possible to use Mini-Batches with Adam optimization?
Adam works best with mid-sized mini batches.
Too small batches can generate too much sampling noise, making Adam less stable than basic stochastic gradient descent.
Too large batches remove the ...
- 33.3k
1
vote
When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
In general, the methods still work even with an infinite amount of data, as long as there are common/reoccuring patterns that a neural network can learn to identify. For example: If you would have ...
- 1,748
1
vote
Accepted
Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?
Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the ...
- 2,754
1
vote
Why is Adam trapped in bad/suspicious local optima after the first few updates?
The authors describe their belief in Section 3:
Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima....
- 1,690
1
vote
Is the choice of the optimiser relevant when doing object detection?
I have experimented with this to a small degree and have not noticed that much of an impact.
To date, Adam appears to give the best results on a variety of image data sets. I have found that "...
- 724
Only top scored, non community-wiki answers of a minimum length are eligible