I developed a custom callback for Keras. Initially, it monitors training accuracy. If on a given epoch the accuracy is below that of the previous epoch it lowers the learning rate by a factor. If for the given epoch the accuracy improves the model's weights are saved. Once the accuracy reaches 95% the callback switches to monitoring validation loss. Again, if the validation loss increases on a given epoch, the learning rate is adjusted downward. If the validation loss decreases on the given epoch the model weights are saved.

This works rather well. However, I got to thinking. When the quantity being monitored at the end of an epoch does not improve, it means you have moved somewhere on a surface in N space (N is the number of trainable parameters) that is further away from the better point on the surface that you had for the previous epoch. Since you have saved the weights from the previous epoch (these weights were "better" than the ones you have at the end of the current epoch) reset the model weights to those of the previous epoch, lower the learning rate and then continue training.

I tried this and it works rather well as expected.

My question is, does this approach converge faster than the standard approach, and does it achieve a more accurate result on the validation set?

I have created a parameter called "dwell" in my callback that I can set to True or False to turn on or off this alternate approach. It is hard to tell if it is converging faster, but on limited testing, I found it usually achieves a lower validation loss.

Are there any papers written on the approach? One thing the callback does for sure is that it allows you to use a larger initial learning rate so it does help the network to converge faster but that is true with or without this alternate training scheme.

  • $\begingroup$ Please, next time, try to use some spaces between paragraphs for easier readability ;) $\endgroup$ – nbro Apr 28 '20 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.