# Why do we explore after we have an accurate estimate of the value function?

Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $$Q$$ values. In that case, why do we still leave a small probability for exploration?

My guess is in the case of a dynamic environment where a bigger reward might pop up in another state. Is my assumption correct?

Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $$Q$$ values. In that case, why do we still leave a small probability for exploration?

It will depend on the goal of the work:

• If the learning algorithm is off-policy (e.g. Q learning), it is normal to continue to explore at a moderate-to-low rate because it can accurately estimate an optimal deterministic target policy from a close-to-optimal stochastic behaviour policy.

• Perhaps it is engineered with a low tolerance and will keep going even when you don't need it to.

• Perhaps the code is for education and run so long that convergence is easily visible. Or for comparison with other methods which really do take that long to converge, and you would like data on the same axis.

• For comparison with other methods for sample efficiency whilst learning and measuring regret (i.e. how much the exploration is costing you).

• When environment is dynamic and could change, then continuous exploration is potentially useful to discover the changes, as you suggest in the question.

If you do really have an ideal agent, then of course you could just stop and say "job done". In practice for more interesting problems, you won't usually get small state spaces and perfect solutions inside 2000 episodes (or ever) - as a result if you are reading tutorials in reinforcement learning, they may just skip this point.

• Ok so due to the randomness of the environment, it’s ideal to always keep exploring in case of new changes to the environment? – Chukwudi Jun 24 at 13:56
• And since small space states are rare, it’s also possible that our space state is close to infinite and therefore even 2000 episodes isn’t enough to have accurate Q values ? – Chukwudi Jun 24 at 13:58
• @Chukwudi: It all depends. However, those are both valid reasons to continue exploring if they are true for your problem. – Neil Slater Jun 24 at 14:15
• Is there anyway I could have a personal chat with you ? – Chukwudi Jun 24 at 14:24
• @Chukwudi: Sure - it's a public chat mind - here - chat.stackexchange.com/rooms/info/109789/… - I am happy to talk about refinements to the question and answer here, but I don't want to take role of persoanl tutor or debug any personal projects etc. – Neil Slater Jun 24 at 14:29

When you are training a system using stochastic gradient descent, your system will converge towards some local minimum. If the local minimum was a good one, we would be fine with it. However, we cannot know how good a found solution is in comparison to other solutions of which we do not know their quality because they have been insufficiently explored. So, continuing to explore is a good way to escape comparatively bad local minima even if training has progressed already for quite a bit.

Besides that, maybe even more importantly towards the end of training, one also wants the system to perform well, i.e. robustly, in the presence of noise and not just under ideal circumstances. So, introducing some randomness, i.e. noise, into the network's policy can also lead to more robust policies being learned since the agent gets trained on how to best recover failure/unforeseen transitions into unexpected states.

• So due to the randomness of the environment( for example a new reward might pop up) it’s ideal to always keep exploring – Chukwudi Jun 24 at 13:57
• Consider the case where an agent has to move across some environment and occasionally slips out or gets displaces by some unforeseen event, e.g. by being displaced by a user (might happen to your lawn mowing robot if you realize that it notoriously misses to visit one particular spot in your garden). Then, the agent will have to no idea how to behave in that situation/state, except for random guessing, if the weights of the network containing the agent's policy have never been trained on coping with such an unforeseen state suddenly being encountered for the first time after training is done). – Daniel B. Jun 24 at 15:03
• Ok so basically we train our agent , it goes to the real world , and at a particular state it was trained to always go up , now if there’s an external interference, where always going up makes it land in a pit, it wouldn’t know what to do and will be confused right, so it’s best we give some randomness – Chukwudi Jun 24 at 15:17
• Yes, because the randomness shall at least somewhat prepare the agent towards how to cope with being in that pit. Maybe the agent hasn't encountered that specific specific situation yet when getting trapped during run time for the first time, but at least it could have encountered comparable situations during training that could enable it to have at least somewhat accurate estimates of Q-values for the novel situation. And that is only achieved if there is at least some exploration constantly throughout training. – Daniel B. Jun 24 at 15:29
• Can we assume the pit it landed in is another state, if it’s another new state it means it has no Idea of what to do in that state since all Q values are 0, so basically giving a chance of exploring is a good thing – Chukwudi Jun 24 at 15:45