1

This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction. For example: Let $L(\theta) = \theta^2 $ and $\theta_0= 3$ Let the subscript n in $\theta_n$ denote the iteration number. Then $\nabla_{\theta}L(\theta_0) = 2*\theta = 2*3 = 6$ For the loss to decrease in this case $\epsilon < 1$ needs to hold ...


Only top voted, non community-wiki answers of a minimum length are eligible