2
$\begingroup$

Why can't we during the first 1000 episodes allow our agent to perform only exploration?

This will give a better chance of covering the entire space state. Then, after the number of episodes, we can then decide to exploit.

$\endgroup$
3
$\begingroup$

Why can’t we during the first 1000 episodes allow our agent perform only exploration

You can do this. It is fine to do so either to learn the value function of a simple random policy, or when performing off-policy updates. It is quite normal when learning an environment from scratch in a safe way - e.g. in simulation - to collect an initial set of data from behaving completely randomly. The amount of this random data varies, and usually the agent will not switch from fully random to fully deterministic based on calculated values as a single step, but will do so gradually.

this will give a better chance of covering the entire space state

That will depend on the nature of the problem. For really simple problems, it may explore the space sufficiently to learn from. However, for many problems it is just a starting point, and not sufficient to cover parts of the space that are of interest in optimal control.

When behaving completely randomly, the agent may take a very long time to complete an episode, and may never complete its first episode. So you could be waiting for a long time to collect data for the first 1000 such episodes. An example of this sort of environment would be a large maze - the agent will move back and forth in the maze, revisiting same parts again and again, where in theory it could already be learning not to repeat its mistakes.

In some environments, behaving completely randomly will result in early failure, and never experiencing postive rewards that are available in the environment. An example of this might be a robot learning to balance on a tightrope and get from one end to the other. It would fall off after a few random actions, gaining very little knowledge for 1000 episodes.

The state space coverage you are looking for ideally should include the optimal path through the space - at least at some point during learning (not necessarily the start). This does not have to appear in one single perfect episode, because the update rules for value functions in reinforcement learning (RL) will eventually allocate the correct values and find this optimal path in the data. However, the collected data does need to include the information about this optimal path amongst all the alternatives so that the methods in RL can evaluate and select it. In simple environments acting randomly may be enough to gain this data, but becomes highly unlikely when the environments are more complex.

then after the number of episodes, we can then decide to exploit

Again this might work for very simple environments, where you have collected enough information through acting randomly to construct a useful value function. However, if acting randomly does not find enough of the optimal path, then the best that exploitation can do is find some local optimum based on the data that was collected.

I suggest you experience this difference for yourself: Set up a toy example environment, and use it to compare different approaches for moving between pure exploration and pure exploitation. You will want to run many experiments (probably 100s for each combination, averaged) to smooth out the randomness, and you can plot the results to see how well each approach learns - e.g. how many time steps (count time steps, not episodes, if you are interested in sample efficiency) it takes for the agent to learn, and whether or not it actually finds the correct optimal behaviour. Bear in mind that the specific results will only apply in your selected environment - so you might also want to do this comparison on a small range of environments.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Ohh, using exploration wouldn’t be able to improve much of the optimal path, I get it, so the Q value estimates wouldn’t be that good since your agent wouldn’t land on those states as often as they do during exploitation, am I right $\endgroup$ – Chukwudi Jun 29 at 10:14
  • $\begingroup$ @Chukwudi: Yes. But as usual this depends heavily on the nature of the environment. I have successfully trained an RL agent to play Tic Tac Toe 100% optimally, learning purely from random moves (for 50,000 episodes), then exploiting the value table that it built. However, the same approach does not work well at all for Connect 4. The way you word your questions, it seems you are looking for hard rules why things should work or not, but RL covers such a large range of problems, that this is not really possible, $\endgroup$ – Neil Slater Jun 29 at 10:17
  • $\begingroup$ Oh I get it now, it’s not a question that can be given a concrete answer, it all depends on the type of environment, what works for one, might not work on the other $\endgroup$ – Chukwudi Jun 29 at 10:20
  • $\begingroup$ @Chukwudi: Yes. There are lots of questions like that in RL. Often when someone here has said "that won't work" to one of your questions, and gives an example why, they mean "that won't work in general, and here's a counter-example". $\endgroup$ – Neil Slater Jun 29 at 10:22
  • $\begingroup$ So the safest way is to gradually decay our exploration rate $\endgroup$ – Chukwudi Jun 29 at 10:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.