0
$\begingroup$

As simple as that. Is there any scenario where the error might increase, if only by a tiny amount, when using SGD (no momentum)?

$\endgroup$
2
$\begingroup$

Yes. Not only that, but error is highly noisy, prone to big spikes and sometimes quite long period of increase before decrease again or stabilize. Often it's even impossible to understand error plot without passing it through smoothing filter, so noisy it is. Specific depend on the problem of cause. It's not only for SGD but for any optimizer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.