Why is exploitation necessary during training?

Question

I have read many blog articles making all kinds of broad analogies to explain the exploration/exploitation trade-off. However, I still can't fully grasp it. On an extremely abstract level, I understand why you would want to "try new things to gain information", but then I don't understand why you would want to "exploit" in training. It seems as though it would be better to keep trying as many things as possible to gain the most information.

What is the value of exploitation during training? Intuitively, I would think you would only want to explore during "training" and only exploit in "testing".

Answer 1

An algorithm that chooses to always explore during training is unlikely to find an optimal policy because it will be employing a more random search as opposed to a directed search. During training, the neural network aims to determine the relation between states or state-action pairs and the reward signal through past experience. If the agent is always exploring during training, it will never use the experience gained from past episodes to influence its training policy and therefore will search more randomly.

Exploitation during training allows the neural net to use its past experience to guide its future actions and avoid random search. There can be many states in which there is an obvious optimal action. After some training, the neural network may be able to quickly learn these states and corresponding optimal actions. By primarily exploiting at those states, the agent will not be wasting training time by exploring suboptimal actions at those states, allowing the agent to focus its exploration on other more uncertain, unexplored, or complex parts of the state space.

For a practical example, consider the original Super Mario Bros game on NES. Let the reward be the number of pixels traveled to the right before Mario loses a life. If Mario is exploring the whole time, it is unlikely that he makes it very far to the right or over many obstacles, let alone to the flagpole. Since it is more rewarding in general to go to the right, Mario's exploitation action at most states is to run to the right. In this manner, Mario will usually run to the right until he reaches an obstacle (e.g. a pipe, pit, staircase, enemy). At that point, Mario may need to explore to overcome the obstacle, but Mario needed to exploit to be able to reach that obstacle in the first place.

Answer 2

Exploitation is important during training to help the network encounter and learn to handle situations that don't occur until the network has successfully navigated other situations.

For example, consider the Atari game Breakout (a common RL benchmark). In this game, the player must move a paddle on the bottom of the screen to bounce a falling ball. The ball accelerates as the game continues, and the network can only get training data with a fast moving ball after successfully exploiting its knowledge of how to play when the ball is moving slowly.

A purely random strategy is possible (and often used for very early training) but only generates data useful for learning the beginning of the game.

Answer 3

Imagine trying to navigate a maze from the outside. Let's say you lose if you get to a dead end, and win if you get to the middle. After some experience by random trials, we know where some dead ends are.

In the future, we should exploit our knowledge and not turn directly into a dead end, as this is simply inefficient. If we do this, we shall find the middle quicker :)

---

This leads to interesting quesitons, such as

'How does the algorithm know if the dead end is always a dead end'.

You might be able to see we might want to tune our randomness rate near a dead end. We might sometimes want to try going down that path, just to be sure, but most of the time lets not bother and go somewhere else. This gives some intuition behind POLICY methods in Reinforcement learning (also see multi-armed bandits if interested) :)

Answer 4

There is an additional factor to consider about exploration/exploitation trade-off, that sometimes applies in addition to the reason in the accepted answer and most other answers here.

Sometimes an agent is required to both act and train itself in a "real" system, or at least one where the rewards are more than just collected training data from a simulation, but also represent actual profits and losses realised by the agent.

This is a common feature of adaptive content display in advertising, which typically uses the simpler k-armed bandit or contextual bandit model - still related to RL, and importantly still affected by the exploration/exploitation trade-off. It is very hard for a machine to model how humans will respond to an advert, so the only reliable measurements are made in production. Each click-through is then real money to someone, so it is important to adapt quickly to incoming data - but due to the variance in results it is also important to still keep testing the non-optimal choice and improve on any early estimates.

In such a scenario, you have to accept some non-optimal returns as the cost of finding the best ones through trial and error. However, it is important to balance this with gaining as much reward as possible whilst training.

So it can be a more important consideration, to obtain best cumulative reward during an ongoing training process, than even finding the optimal policy. That means taking care to balance exploitation and exploration, often strongly favouring exploitation after a relatively short high exploration phase.

Answer 5

If you explore too much, you waste your time (among other resources.) You will probably exhaust your resources before you learn anything meaningful.

Let's say your goal is to learn as much about Star Wars as possible within a library. If you fully explore, you just pick books at random.

Exploitation might look something like "pick most of your books within the Sci-Fi section", or "choose books with lightsabers on the cover" or "books with 'Star Wars' in the title" because that's where you have found relevant information in the past.

Answer 6

In supervised ML there is no exploration and exploitation.

In reinforcement learning, the agent in each step has many choices.

So the agent can exploit, meaning gaining the highest reward known to him from the next move. Or explore, trying to get a better long-term benefit by trying a different move.