Hot answers tagged

7

I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. Also, using bold letters to represent vectors. In that notation, the update rule for basic gradient descent would be written as: $$\mathbf{w} \leftarrow \mathbf{...


1

Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the explosion of the gradients magnitude and leads to more steady convergence. Adam is an adaptive optimizer with momentum and division by the weighted sum of gradients on ...


1

I have experimented with this to a small degree and have not noticed that much of an impact. To date, Adam appears to give the best results on a variety of image data sets. I have found that "adjusting" the learning rate during training is an effective means of improving model performance and has more impact than the selection of the optimizer. ...


Only top voted, non community-wiki answers of a minimum length are eligible