As far as I understand, RL is a process that can be divided into 2 stages:

  1. Exploring a wide range of paths (acting randomly)

  2. Refining the current optimal paths (revolving around actions with a so-far most promising score estimate)

Completing 1. too quickly results in a network that just doesn't spot the best combination of actions, especially if rewards are sparse. "Refining" then has little benefit, since the network will tend to choose between unlucky estimates it observed so far, and will specialise in those.

On the other hand, finishing 2. too quickly results in a network that might have encountered the best combination, but never got time to refine these "good trajectories". Thus its estimates of scores along these "good trajectories" are rather poor and inaccurate, so again the network will fear to select and specialize those, because they might have a low (inaccurate) estimate.

Why not to give both 1. and 2. the maximum time possible?

In other words, instead of gradually annealing the $\epsilon$ coefficient (in the $\epsilon$-greedy) down to a low value, why not to always have it as a step function?

For example, train 50% of iterations with a value of 1 (acting completely randomly), and for the second half of training with the value of 0.05, etc (very greedy). Well, 50% is a random guess, could be adjusted manually, as needed. The most important part is this "step function".

To me, always using such a "step" function would instantly reveal if the initial random search was not long enough. Perhaps there is a disadvantage of such a step curve?

So far, I got the impression that annealing is a gradual process. To me, it seems that when using gradual annealing it might not be evident if the neural network (e.q. in DQN or DQRNN) learns poorly because of the mentioned issue or something else.

Is there some literature exploring this?

There is a paper Noisy Networks for Exploration, but it proposes another approach that removes the $\epsilon$ hyperparameter. My question is different, specifically, about this $\epsilon$.

  • $\begingroup$ This seems like something that would be easy to explore in an experiment. Pick a few simple environments, and compare a step function change to $\epsilon$ with gradual change. You will need to tune both approaches (each has a few hyper-parameters) to compare best with best, but experiment on something simple might only take a few days $\endgroup$ Jul 11 '19 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.