Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small probability for exploration?
My guess is in the case of a dynamic environment where a bigger reward might pop up in another state. Is my assumption correct?
Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small probability for exploration?
It will depend on the goal of the work:
If the learning algorithm is off-policy (e.g. Q learning), it is normal to continue to explore at a moderate-to-low rate because it can accurately estimate an optimal deterministic target policy from a close-to-optimal stochastic behaviour policy.
Perhaps it is engineered with a low tolerance and will keep going even when you don't need it to.
Perhaps the code is for education and run so long that convergence is easily visible. Or for comparison with other methods which really do take that long to converge, and you would like data on the same axis.
For comparison with other methods for sample efficiency whilst learning and measuring regret (i.e. how much the exploration is costing you).
When environment is dynamic and could change, then continuous exploration is potentially useful to discover the changes, as you suggest in the question.
If you do really have an ideal agent, then of course you could just stop and say "job done". In practice for more interesting problems, you won't usually get small state spaces and perfect solutions inside 2000 episodes (or ever) - as a result if you are reading tutorials in reinforcement learning, they may just skip this point.
When you are training a system using stochastic gradient descent, your system will converge towards some local minimum. If the local minimum was a good one, we would be fine with it. However, we cannot know how good a found solution is in comparison to other solutions of which we do not know their quality because they have been insufficiently explored. So, continuing to explore is a good way to escape comparatively bad local minima even if training has progressed already for quite a bit.
Besides that, maybe even more importantly towards the end of training, one also wants the system to perform well, i.e. robustly, in the presence of noise and not just under ideal circumstances. So, introducing some randomness, i.e. noise, into the network's policy can also lead to more robust policies being learned since the agent gets trained on how to best recover failure/unforeseen transitions into unexpected states.
