9 votes
Accepted

What is the formula for the momentum and Adam optimisers?

I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. ...
Neil Slater's user avatar
3 votes
Accepted

Is it possible to use Mini-Batches with Adam optimization?

Adam works best with mid-sized mini batches. Too small batches can generate too much sampling noise, making Adam less stable than basic stochastic gradient descent. Too large batches remove the ...
Neil Slater's user avatar
1 vote

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?

In general, the methods still work even with an infinite amount of data, as long as there are common/reoccuring patterns that a neural network can learn to identify. For example: If you would have ...
Chillston's user avatar
  • 1,694
1 vote
Accepted

Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?

Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the ...
spiridon_the_sun_rotator's user avatar
1 vote

Why is Adam trapped in bad/suspicious local optima after the first few updates?

The authors describe their belief in Section 3: Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima....
user3667125's user avatar
  • 1,560
1 vote

Is the choice of the optimiser relevant when doing object detection?

I have experimented with this to a small degree and have not noticed that much of an impact. To date, Adam appears to give the best results on a variety of image data sets. I have found that "...
Gerry P's user avatar
  • 714

Only top scored, non community-wiki answers of a minimum length are eligible