1
$\begingroup$

I'm currently modeling DQN in Reinforcement Learning. My question is: what are the best practices related to Boltzmann Exploration? My current thoughts are: (1) Let the temperature decay through training and finally stop at 0.01, when the method will always select the best practice, with almost no randomness. (2) Standardize the predicted Q values before feeding into the softmax function.

Currently, I'm using (2), and the reward is suffering from high variance. I'm wondering whether it has something to do with the exploration method?

$\endgroup$
1
$\begingroup$

Do you have to use Boltzmann exploration, strictly? There is a modification for Boltzmann exploration called Mellow-max. It, basically, provides an adaptive temperature for Boltzmann exploration.

Here is the link for the paper for tuning mellow-max with deep reinforcement learning (DQN is often mentioned): http://cs.brown.edu/people/gdk/pubs/tuning_mellowmax_drlw.pdf

Here is the link for mellow-max implemented with SARSA (I recommend reading this first, to get an understanding of mellow-max): https://arxiv.org/pdf/1612.05628.pdf

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks and upvoted for your suggestion. $\endgroup$ – o_yeah Jun 26 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.