9
votes
Accepted
Why is exploitation necessary during training?
An algorithm that chooses to always explore during training is unlikely to find an optimal policy because it will be employing a more random search as opposed to a directed search. During training, ...
- 1,122
7
votes
Accepted
Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
- 471
5
votes
Why is exploitation necessary during training?
Exploitation is important during training to help the network encounter and learn to handle situations that don't occur until the network has successfully navigated other situations.
For example, ...
- 471
5
votes
Accepted
In Q-learning, wouldn't it be better to simply iterate through all possible states?
If your algorithm is executed multiple (or enough) times using an outer loop, it would converge to similar results as Q-learning would with $\gamma = 0$ (as you don't look what is the expected future ...
- 424
5
votes
Accepted
What is the meaning of "exploration" in reinforcement and supervised learning?
In reinforcement learning, exploration has a specific meaning, which is in contrast with the meaning of exploitation, hence the so-called exploration-exploitation dilemma (or trade-off). You explore ...
- 37k
4
votes
Accepted
What is the purpose of the actor in actor-critic algorithms?
For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms?
In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used ...
- 26.5k
3
votes
Accepted
Does eligibility traces and epsilon-greedy do the same task in different ways?
Epsilon-greedy is one method of making an agent explore the state space to ensure that the agent doesn't settle on a sub-optimal policy. By taking random actions, even with a small probability, the ...
- 3,697
3
votes
Accepted
Which policy has to be followed by a player while construction of its own Q-table?
I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA).
Q learning is an off policy algorithm, meaning that the Q values can be ...
- 1,071
3
votes
Accepted
Why is 100% exploration bad during the learning stage in reinforcement learning?
Why can’t we during the first 1000 episodes allow our agent perform only exploration
You can do this. It is fine to do so either to learn the value function of a simple random policy, or when ...
- 26.5k
2
votes
Should I use exploration strategy in Policy Gradient algorithms?
Neil Slater's answer is very nice, but I have a couple more suggestions:
You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for ...
- 1,071
2
votes
Accepted
Should I use exploration strategy in Policy Gradient algorithms?
I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.
Yes, ...
- 26.5k
2
votes
Why is it not advisable to have a 100 percent exploration rate?
No - imagine if you were playing an Atari game and took completely random actions. Your games would not last very long and you would never get to experience all of the state space because the game ...
- 4,400
2
votes
Why can't we fully exploit the environment after the first episode in Q-learning?
BlueTurtle's answer is good, but I'd like to add something.
Your question realistically has nothing to do with Q Learning, in fact, you can ask the same thing about just about any RL algorithm. In ...
- 1,071
2
votes
In Q-learning, wouldn't it be better to simply iterate through all possible states?
In short, yes, provided that you have a small number of states.
In pretty much any real system, the number of states is much higher than you could ever hope to explore exhaustively in any reasonable ...
- 583
2
votes
Can we stop training as soon as epsilon is small?
How much the $Q$-values change does not depend on the value of $\epsilon$, rather the value of $\epsilon$ dictates how likely you are to take a random action and thus take an action that could give ...
- 4,400
2
votes
Are RL algorithms suppose to keep learning?
This depends on the setting. Ongoing learning that never ends is a feature of settings where one or both of the following is true:
There is little existing available data or experience when ...
- 26.5k
2
votes
Why is exploitation necessary during training?
If you explore too much, you waste your time (among other resources.) You will probably exhaust your resources before you learn anything meaningful.
Let's say your goal is to learn as much about Star ...
- 121
2
votes
Why is exploitation necessary during training?
There is an additional factor to consider about exploration/exploitation trade-off, that sometimes applies in addition to the reason in the accepted answer and most other answers here.
Sometimes an ...
- 26.5k
2
votes
Why is exploitation necessary during training?
Imagine trying to navigate a maze from the outside. Let's say you lose if you get to a dead end, and win if you get to the middle. After some experience by random trials, we know where some dead ends ...
- 21
1
vote
How to reduce the number of episodes before the agent learns in this game?
TLDR: Simplify your agent.
Context:
As you've noticed, it's not a hard game and it does not require a complex policy.
You'd need a single neuron to solve it perfectly: ...
- 824
1
vote
Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
I think there is an implicit notion of it in dynamic programming; say, if you have to make some sort of search over a subset of a state space and you are deciding whether to use BFS, breath first ...
- 121
1
vote
Accepted
What is the optimal exploration-exploitation trade-off in Q*bert?
I can spot three, maybe four, things in your implementation that could be contributing to incomplete learning that you are observing.
More exploration in long term
I think you have correctly ...
- 26.5k
1
vote
Accepted
Why do some DQN implementations not require random exploration but instead emulate all actions?
The example that you linked is using a model (emulation) in order to look ahead at all possible actions from any state. It essentially explores off-policy and offline using that model. This is not an ...
- 26.5k
1
vote
Accepted
Why can't we fully exploit the environment after the first episode in Q-learning?
You can't guarantee that you have taken every action from every state, even with 1000 time steps. There would be multiple outcomes:
The episode terminates, either by success or failure before the ...
- 160
1
vote
Accepted
Can tabular Q-learning converge even if it doesn't explore all state-action pairs?
In the tabular case, then the Q table will only converge if you have walked around the whole of the table. Note that to guarantee convergence we need $\sum\limits_{n=1}^{\infty}\alpha_n(a) = \infty$ ...
- 4,400
1
vote
Should I just use exploitation after I have trained the Q agent?
Once you have estimated the $Q$ function, you can derive the policy from it in different ways. For example, you can act greedily with respect to it (see this answer), which can be formally denoted as
...
- 37k
Only top scored, non community-wiki answers of a minimum length are eligible