2 votes

Is learning rate the only reason for training loss oscillation after few epochs?

The loss graph indicates that the model converged to a local minimum, already after a few epochs, and the weights start to oscillate around it. The learning rate is surely responsible for it, but it's ...
1 vote

Why can the learning rate make the loss increase in stochastic gradient descent?

This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction. For example: Let $L(\theta) = \theta^2 $ and $\theta_0= 3$ Let the subscript n in $\...
  • 349

Only top scored, non community-wiki answers of a minimum length are eligible