Why is 100% exploration bad during the learning stage in reinforcement learning?

Question

Why can't we during the first 1000 episodes allow our agent to perform only exploration?

This will give a better chance of covering the entire space state. Then, after the number of episodes, we can then decide to exploit.

Answer 1

Why can’t we during the first 1000 episodes allow our agent perform only exploration

You can do this. It is fine to do so either to learn the value function of a simple random policy, or when performing off-policy updates. It is quite normal when learning an environment from scratch in a safe way - e.g. in simulation - to collect an initial set of data from behaving completely randomly. The amount of this random data varies, and usually the agent will not switch from fully random to fully deterministic based on calculated values as a single step, but will do so gradually.

this will give a better chance of covering the entire space state

That will depend on the nature of the problem. For really simple problems, it may explore the space sufficiently to learn from. However, for many problems it is just a starting point, and not sufficient to cover parts of the space that are of interest in optimal control.

When behaving completely randomly, the agent may take a very long time to complete an episode, and may never complete its first episode. So you could be waiting for a long time to collect data for the first 1000 such episodes. An example of this sort of environment would be a large maze - the agent will move back and forth in the maze, revisiting same parts again and again, where in theory it could already be learning not to repeat its mistakes.

In some environments, behaving completely randomly will result in early failure, and never experiencing postive rewards that are available in the environment. An example of this might be a robot learning to balance on a tightrope and get from one end to the other. It would fall off after a few random actions, gaining very little knowledge for 1000 episodes.

The state space coverage you are looking for ideally should include the optimal path through the space - at least at some point during learning (not necessarily the start). This does not have to appear in one single perfect episode, because the update rules for value functions in reinforcement learning (RL) will eventually allocate the correct values and find this optimal path in the data. However, the collected data does need to include the information about this optimal path amongst all the alternatives so that the methods in RL can evaluate and select it. In simple environments acting randomly may be enough to gain this data, but becomes highly unlikely when the environments are more complex.

then after the number of episodes, we can then decide to exploit

Again this might work for very simple environments, where you have collected enough information through acting randomly to construct a useful value function. However, if acting randomly does not find enough of the optimal path, then the best that exploitation can do is find some local optimum based on the data that was collected.

I suggest you experience this difference for yourself: Set up a toy example environment, and use it to compare different approaches for moving between pure exploration and pure exploitation. You will want to run many experiments (probably 100s for each combination, averaged) to smooth out the randomness, and you can plot the results to see how well each approach learns - e.g. how many time steps (count time steps, not episodes, if you are interested in sample efficiency) it takes for the agent to learn, and whether or not it actually finds the correct optimal behaviour. Bear in mind that the specific results will only apply in your selected environment - so you might also want to do this comparison on a small range of environments.