During the learning phase, why don't we have a 100% exploration rate, to allow our agent to fully explore our environment and update the Q values, then during testing we bring in exploitation? Does that make more sense than decaying the exploration rate?


No - imagine if you were playing an Atari game and took completely random actions. Your games would not last very long and you would never get to experience all of the state space because the game would end too soon. This is why you need to combine exploration and exploitation to fully explore the state space.

  • $\begingroup$ But you get my point, even if the game ends quickly, suppose you have a lot of episodes, eventually all space states would be explored , then when we want to test it , we bring in exploitation $\endgroup$ – Chukwudi Jun 26 '20 at 17:05
  • $\begingroup$ And also after full training, why is it not advisable to turn off exploration, why do we still leave room for small exploration, my guess is in the case of a dynamic environment where new states might appear $\endgroup$ – Chukwudi Jun 26 '20 at 17:07
  • $\begingroup$ @Chukwudi no, if you took random actions in an Atari game you would never win the game in a feasible amount of episodes. Maybe theoretically in the limit you would, but remember that we have limited compute power, and thus you need to exploit the good actions so that we can keep learning from the good actions by backing up the Q function for these state-action values. However, exploration must be maintained to some degree to ensure that we aren't missing any better actions. $\endgroup$ – David Ireland Jun 26 '20 at 18:13
  • $\begingroup$ What if we’re sure we’ve covered every state and the Q values are very accurate, assuming our space state is small, the what would be the use of still exploring $\endgroup$ – Chukwudi Jun 26 '20 at 18:20
  • $\begingroup$ In terms of making Q-learning scale well, relatively, in environments with a large state space, there must be exploration and exploitation since the aim of Q-learning is to find an optimum policy to traverse an optimum path from start to goal in an MDP. So exploration helps to check if there is a better action to take given a state and set of actions and then leading to a better policy and exploitation allows the agent to find it in a feasible amount of time. Note some environments can be very huge e.g. (256 ^ 128) states. $\endgroup$ – rert588 Jun 26 '20 at 18:27

While theoretically you can do something like this if you're very confident you'll cover most of the state space in exploration, this is still a suboptimal strategy. Even in the case of multi-armed bandits, this strategy can be much less sample efficient than $\epsilon$-greedy, and exploration is much easier in this case.

So, even if your strategy miraculously works on a decently sized MDP, it'll be worse than combining exploration and exploitation.

  • $\begingroup$ Why do you say it’ll be worse, the whole point of me using just exploration is to cover the whole states, and improve my policy $\endgroup$ – Chukwudi Jun 28 '20 at 1:02
  • $\begingroup$ Because if you're doing 100% exploration, you'll be revisiting states very often as well. Furthermore of your state space is large, covering the whole state space is intractable. In theory, as long as you explore forever (like in e-greedy for instance), you can still cover the whole state space, but you'll be exploiting way more and incurring less regret. $\endgroup$ – harwiltz Jun 28 '20 at 14:47
  • $\begingroup$ But I need a concrete answer as to why we need to combine exploration and exploitation, because to me, exploration 100% would cover the entire space state After like 1000 episodes, them we can start exploiting $\endgroup$ – Chukwudi Jun 29 '20 at 2:04
  • $\begingroup$ Unless your state space is very small, you won't cover 100% of the state space with full exploration. Especially if the environment is difficult to navigate, without any exploitation your agent will probably just keep failing immediately (think CartPole for instance). For a concrete answer, like I said, look into the Multi-Armed Bandit literature (even Sutton and Barto). You can prove that your strategy does not achieve optimal regret. $\endgroup$ – harwiltz Jun 29 '20 at 14:21
  • $\begingroup$ Someone used a robot balancing on a rope, if I use 100% exploration, then the random actions will make the episodes shorter, and I will never be able to explore the environment in those number of episodes $\endgroup$ – Chukwudi Jun 29 '20 at 19:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.