8 votes
Accepted

Why is exploitation necessary during training?

An algorithm that chooses to always explore during training is unlikely to find an optimal policy because it will be employing a more random search as opposed to a directed search. During training, ...
user avatar
  • 928
7 votes
Accepted

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
user avatar
5 votes

Why is exploitation necessary during training?

Exploitation is important during training to help the network encounter and learn to handle situations that don't occur until the network has successfully navigated other situations. For example, ...
user avatar
5 votes
Accepted

What is the meaning of "exploration" in reinforcement and supervised learning?

In reinforcement learning, exploration has a specific meaning, which is in contrast with the meaning of exploitation, hence the so-called exploration-exploitation dilemma (or trade-off). You explore ...
user avatar
  • 33.8k
5 votes
Accepted

In Q-learning, wouldn't it be better to simply iterate through all possible states?

If your algorithm is executed multiple (or enough) times using an outer loop, it would converge to similar results as Q-learning would with $\gamma = 0$ (as you don't look what is the expected future ...
user avatar
  • 424
4 votes
Accepted

What is the purpose of the actor in actor-critic algorithms?

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms? In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used ...
user avatar
  • 23.9k
3 votes
Accepted

Why is 100% exploration bad during the learning stage in reinforcement learning?

Why can’t we during the first 1000 episodes allow our agent perform only exploration You can do this. It is fine to do so either to learn the value function of a simple random policy, or when ...
user avatar
  • 23.9k
3 votes
Accepted

Does eligibility traces and epsilon-greedy do the same task in different ways?

Epsilon-greedy is one method of making an agent explore the state space to ensure that the agent doesn't settle on a sub-optimal policy. By taking random actions, even with a small probability, the ...
user avatar
3 votes
Accepted

Which policy has to be followed by a player while construction of its own Q-table?

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA). Q learning is an off policy algorithm, meaning that the Q values can be ...
user avatar
  • 961
2 votes

In Q-learning, wouldn't it be better to simply iterate through all possible states?

In short, yes, provided that you have a small number of states. In pretty much any real system, the number of states is much higher than you could ever hope to explore exhaustively in any reasonable ...
user avatar
  • 583
2 votes

Why is exploitation necessary during training?

If you explore too much, you waste your time (among other resources.) You will probably exhaust your resources before you learn anything meaningful. Let's say your goal is to learn as much about Star ...
user avatar
2 votes

Why is exploitation necessary during training?

There is an additional factor to consider about exploration/exploitation trade-off, that sometimes applies in addition to the reason in the accepted answer and most other answers here. Sometimes an ...
user avatar
  • 23.9k
2 votes

Why is exploitation necessary during training?

Imagine trying to navigate a maze from the outside. Let's say you lose if you get to a dead end, and win if you get to the middle. After some experience by random trials, we know where some dead ends ...
user avatar
  • 21
2 votes
Accepted

Should I use exploration strategy in Policy Gradient algorithms?

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state. Yes, ...
user avatar
  • 23.9k
2 votes

Why is it not advisable to have a 100 percent exploration rate?

No - imagine if you were playing an Atari game and took completely random actions. Your games would not last very long and you would never get to experience all of the state space because the game ...
user avatar
2 votes

Why can't we fully exploit the environment after the first episode in Q-learning?

BlueTurtle's answer is good, but I'd like to add something. Your question realistically has nothing to do with Q Learning, in fact, you can ask the same thing about just about any RL algorithm. In ...
user avatar
  • 961
2 votes

Can we stop training as soon as epsilon is small?

How much the $Q$-values change does not depend on the value of $\epsilon$, rather the value of $\epsilon$ dictates how likely you are to take a random action and thus take an action that could give ...
user avatar
1 vote

How to reduce the number of episodes before the agent learns in this game?

TLDR: Simplify your agent. Context: As you've noticed, it's not a hard game and it does not require a complex policy. You'd need a single neuron to solve it perfectly: ...
user avatar
1 vote

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?

I think there is an implicit notion of it in dynamic programming; say, if you have to make some sort of search over a subset of a state space and you are deciding whether to use BFS, breath first ...
user avatar
1 vote
Accepted

What is the optimal exploration-exploitation trade-off in Q*bert?

I can spot three, maybe four, things in your implementation that could be contributing to incomplete learning that you are observing. More exploration in long term I think you have correctly ...
user avatar
  • 23.9k
1 vote
Accepted

Why do some DQN implementations not require random exploration but instead emulate all actions?

The example that you linked is using a model (emulation) in order to look ahead at all possible actions from any state. It essentially explores off-policy and offline using that model. This is not an ...
user avatar
  • 23.9k
1 vote
Accepted

Why can't we fully exploit the environment after the first episode in Q-learning?

You can't guarantee that you have taken every action from every state, even with 1000 time steps. There would be multiple outcomes: The episode terminates, either by success or failure before the ...
user avatar
1 vote

Should I use exploration strategy in Policy Gradient algorithms?

Neil Slater's answer is very nice, but I have a couple more suggestions: You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for ...
user avatar
  • 961
1 vote
Accepted

Can tabular Q-learning converge even if it doesn't explore all state-action pairs?

In the tabular case, then the Q table will only converge if you have walked around the whole of the table. Note that to guarantee convergence we need $\sum\limits_{n=1}^{\infty}\alpha_n(a) = \infty$ ...
user avatar
1 vote

Should I just use exploitation after I have trained the Q agent?

Once you have estimated the $Q$ function, you can derive the policy from it in different ways. For example, you can act greedily with respect to it (see this answer), which can be formally denoted as ...
user avatar
  • 33.8k

Only top scored, non community-wiki answers of a minimum length are eligible