0
$\begingroup$

Regarding of DQN, or DQRNN, (reinforcement learning)

To me, RL is a process that can be divided into 2 stages:

  1. Exploring wide range of paths (acting randomly)
  2. Refining the current optimal paths (revolving around actions with a so-far most promising score estimate)

Completing 1. too quickly results in a network that just doesn't spot the best combination of actions, especially if rewards are sparse. "Refining" then has little benefit, since the network will tend to choose between unlucky estimates it observed so far, and will specialise in those.

On the other hand, finishing 2. too quickly results in a network that might have encountered the best combination, but never got time to refine these "good trajectories". Thus its estimates of scores along these "good trajectories" are rather poor and inacurate, so again the network will fear to select and specialize those, because they might have low (innacurate) estimate.

Question:

Why not to give both 1. and 2. the maximum time possible?

In other words, instead of gradually annealing epsilon-greedy coefficient down to a low value, why not to always have it as a step function?

For example, train 50% of iterations with a value of 1 (acting completely randomly), and for the second half of training with the value of 0.05, etc (very greedy). Well, 50% is a random guess, could be adjusted manually, as needed. The most important part is this "step function".

To me, always using such a "step" function would instantly reveal if the initial random search was not long enough. Perhaps there is a disadvantage of such a step curve?

So far I got the impression that annealing is a gradual process. To me it seems that when using gradual annealing it might not be evident if network learns poorly because of the mentioned issue or something else.

Is there some literature exploring this? There is a paper "Noisy Networks for Exploration", but it proposes another approach which removes the $\epsilon$-greedy hyperparameter. My question is different, - specifically about this $\epsilon$-greedy.

$\endgroup$
  • $\begingroup$ This seems like something that would be easy to explore in an experiment. Pick a few simple environments, and compare a step function change to $\epsilon$ with gradual change. You will need to tune both approaches (each has a few hyper-parameters) to compare best with best, but experiment on something simple might only take a few days $\endgroup$ – Neil Slater Jul 11 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.