Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ @16Aghnar in what scenario would I use a lower learning rate? I've looked at the ATARI papers and games like Super Mario Bros. They used learning rates of 0.00025. Is it because we want it to not get hooked onto the same decision again and again? As in, with a higher learning rate, it would assume action x would be best. I hope I made sense! $\endgroup$– rtzCommented Sep 12, 2018 at 0:12
-
1$\begingroup$ So, a lower LR means a slower convergence but an improved asymptote (limit of the learning curve). So tuning it depends on the time you have, and also on your model. You can begin with, for example, 0.001, see the learning curve, and if you reach quickly the asymptote, you can try with a lower LR, see again the learning curve, and so on. (and notice that the 0.98 - 0.997 values I mentioned in my answer are for the LR decay, not for the LR) $\endgroup$– 16AghnarCommented Sep 12, 2018 at 8:07
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay: they play at the same scale (the number of successive batches you'll train your model on). But if I refine the analysis, I would rather say that it's a reason to choose them in the same range, i.e. close to 1, but not specifically at the same number.
For LR decay, people often choose it very close to one (which can mean really different things like 0.98 or 0.997), because it plays on a large scale, and you don't want the LR to disappear too brutally.
However, the choice of ER decay can have more variation from model to model. It depends on the initial value of ER (you don't wanna decay fastly ER if your ER is initially low), and also to the "learning speed" of your model: if your model learns efficiently at the beginning, you could want to fastly decrease ER in order to reduce the noise on the action, supposing that you did enough exploration at the beginning (but I think this last opinion is more controversial). You can find an interesting paper here, where the author tries different ER decay and finds out that 0.99 is the best, for CartPole environment.
