Hot answers tagged

9 votes

What is the formula for the momentum and Adam optimisers?

I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. ...
  • 24.6k
1 vote

Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?

Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the ...
1 vote

Is the choice of the optimiser relevant when doing object detection?

I have experimented with this to a small degree and have not noticed that much of an impact. To date, Adam appears to give the best results on a variety of image data sets. I have found that "...
  • 664

Only top scored, non community-wiki answers of a minimum length are eligible