6
$\begingroup$

Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?

$\endgroup$
  • 2
    $\begingroup$ I made a major edit of my answer. I misanderstood your question : reading too fast, I took exploration rate for the reward discount factor. Big mistake, and it's mine ! I apologize. So I changed the analysis for ER :) $\endgroup$ – 16Aghnar Sep 11 '18 at 18:57
  • $\begingroup$ @16Aghnar in what scenario would I use a lower learning rate? I've looked at the ATARI papers and games like Super Mario Bros. They used learning rates of 0.00025. Is it because we want it to not get hooked onto the same decision again and again? As in, with a higher learning rate, it would assume action x would be best. I hope I made sense! $\endgroup$ – rtz Sep 12 '18 at 0:12
  • 1
    $\begingroup$ So, a lower LR means a slower convergence but an improved asymptote (limit of the learning curve). So tuning it depends on the time you have, and also on your model. You can begin with, for example, 0.001, see the learning curve, and if you reach quickly the asymptote, you can try with a lower LR, see again the learning curve, and so on. (and notice that the 0.98 - 0.997 values I mentioned in my answer are for the LR decay, not for the LR) $\endgroup$ – 16Aghnar Sep 12 '18 at 8:07
6
$\begingroup$

Welcome to AI.SE ! First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay : they play at the same scale (the number of successives batches you'll train your model on). But if I refine the analysis, I would rather say that it's a reason to choose them in the same range, i.e. close to 1, but not specially at the same number.

  • For LR decay, people often choose it very close to one (which can mean really different things like 0.98 or 0.997), because it plays on a large scale, an you don't want the LR to disappear to brutally.

  • However, the choice of ER decay can have more variation from model to model. It depends on the initial value of ER (you don't wanna decay fastly ER if you ER is initially low), and also to the "learning speed" of your model : if your model learns efficiently at the beginning, you could want to fastly decrease ER in order to reduce the noise on the action, supposing that you did enough exploration at the beginning (but I think this last opinion is more controversial). You can find an interesting paper here, where the author tries different ER decay and finds out that 0.99 is the best, for CartPole environment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.