9 votes
Accepted

Why is exploitation necessary during training?

An algorithm that chooses to always explore during training is unlikely to find an optimal policy because it will be employing a more random search as opposed to a directed search. During training, ...
DeepQZero's user avatar
  • 1,357
7 votes
Accepted

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
Lee Reeves's user avatar
5 votes

Why is exploitation necessary during training?

Exploitation is important during training to help the network encounter and learn to handle situations that don't occur until the network has successfully navigated other situations. For example, ...
Lee Reeves's user avatar
5 votes
Accepted

In Q-learning, wouldn't it be better to simply iterate through all possible states?

If your algorithm is executed multiple (or enough) times using an outer loop, it would converge to similar results as Q-learning would with $\gamma = 0$ (as you don't look what is the expected future ...
kirua's user avatar
  • 424
5 votes
Accepted

What is the meaning of "exploration" in reinforcement and supervised learning?

In reinforcement learning, exploration has a specific meaning, which is in contrast with the meaning of exploitation, hence the so-called exploration-exploitation dilemma (or trade-off). You explore ...
nbro's user avatar
  • 40.2k
4 votes
Accepted

What is the purpose of the actor in actor-critic algorithms?

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms? In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used ...
Neil Slater's user avatar
  • 31.7k
3 votes
Accepted

Why is 100% exploration bad during the learning stage in reinforcement learning?

Why can’t we during the first 1000 episodes allow our agent perform only exploration You can do this. It is fine to do so either to learn the value function of a simple random policy, or when ...
Neil Slater's user avatar
  • 31.7k
3 votes
Accepted

Which policy has to be followed by a player while construction of its own Q-table?

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA). Q learning is an off policy algorithm, meaning that the Q values can be ...
harwiltz's user avatar
  • 1,126
3 votes
Accepted

Does eligibility traces and epsilon-greedy do the same task in different ways?

Epsilon-greedy is one method of making an agent explore the state space to ensure that the agent doesn't settle on a sub-optimal policy. By taking random actions, even with a small probability, the ...
Jaden Travnik's user avatar
2 votes

Should I use exploration strategy in Policy Gradient algorithms?

Neil Slater's answer is very nice, but I have a couple more suggestions: You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for ...
harwiltz's user avatar
  • 1,126
2 votes
Accepted

Should I use exploration strategy in Policy Gradient algorithms?

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state. Yes, ...
Neil Slater's user avatar
  • 31.7k
2 votes

Why can't we fully exploit the environment after the first episode in Q-learning?

BlueTurtle's answer is good, but I'd like to add something. Your question realistically has nothing to do with Q Learning, in fact, you can ask the same thing about just about any RL algorithm. In ...
harwiltz's user avatar
  • 1,126
2 votes

In Q-learning, wouldn't it be better to simply iterate through all possible states?

In short, yes, provided that you have a small number of states. In pretty much any real system, the number of states is much higher than you could ever hope to explore exhaustively in any reasonable ...
DrMcCleod's user avatar
  • 603
2 votes

Can we stop training as soon as epsilon is small?

How much the $Q$-values change does not depend on the value of $\epsilon$, rather the value of $\epsilon$ dictates how likely you are to take a random action and thus take an action that could give ...
David's user avatar
  • 4,770
2 votes

Why is it not advisable to have a 100 percent exploration rate?

No - imagine if you were playing an Atari game and took completely random actions. Your games would not last very long and you would never get to experience all of the state space because the game ...
David's user avatar
  • 4,770
2 votes

Why is exploitation necessary during training?

If you explore too much, you waste your time (among other resources.) You will probably exhaust your resources before you learn anything meaningful. Let's say your goal is to learn as much about Star ...
knallfrosch's user avatar
2 votes

Why is exploitation necessary during training?

There is an additional factor to consider about exploration/exploitation trade-off, that sometimes applies in addition to the reason in the accepted answer and most other answers here. Sometimes an ...
Neil Slater's user avatar
  • 31.7k
2 votes

Why is exploitation necessary during training?

Imagine trying to navigate a maze from the outside. Let's say you lose if you get to a dead end, and win if you get to the middle. After some experience by random trials, we know where some dead ends ...
Charles's user avatar
  • 21
2 votes

Are RL algorithms suppose to keep learning?

This depends on the setting. Ongoing learning that never ends is a feature of settings where one or both of the following is true: There is little existing available data or experience when ...
Neil Slater's user avatar
  • 31.7k
1 vote

Practical differences between the different RL architectures

In theory on-policy algorithms like PPO and actor-critic should perform better than off-policy algorithms such as DQN. The downside is that you will need to collect new data once the policy is updated,...
pi-tau's user avatar
  • 702
1 vote
Accepted

Methods for sequential decision optimization problem with nonlinear bayesian reward function

One possible alternative approach is incorporating deep reinforcement learning (DRL) techniques. These techniques are sequential since they incorporate lookahead, and they are designed to attempt to ...
DeepQZero's user avatar
  • 1,357
1 vote
Accepted

In which community does using a Bayesian regression model as a reward function with exploration vs. exploitation challenges fall under?

One community that has very recently been attacking problems of the type posed by your question is the Bayesian sequential optimal experimental design (Bayesian sOED) community. The Bayesian sOED ...
DeepQZero's user avatar
  • 1,357
1 vote

Metrics to compare the exploraation of RL Algorithms

For 2D (labyrinth-like) tasks the measure is usually percentage of the space that was covered. Another option is distance moved (in any direction). Once you have an objective measure for exploration, ...
maxy's user avatar
  • 223
1 vote

How to reduce the number of episodes before the agent learns in this game?

TLDR: Simplify your agent. Context: As you've noticed, it's not a hard game and it does not require a complex policy. You'd need a single neuron to solve it perfectly: ...
Andre Goulart's user avatar
1 vote

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?

I think there is an implicit notion of it in dynamic programming; say, if you have to make some sort of search over a subset of a state space and you are deciding whether to use BFS, breath first ...
João Schapke's user avatar
1 vote
Accepted

What is the optimal exploration-exploitation trade-off in Q*bert?

I can spot three, maybe four, things in your implementation that could be contributing to incomplete learning that you are observing. More exploration in long term I think you have correctly ...
Neil Slater's user avatar
  • 31.7k
1 vote
Accepted

Why do some DQN implementations not require random exploration but instead emulate all actions?

The example that you linked is using a model (emulation) in order to look ahead at all possible actions from any state. It essentially explores off-policy and offline using that model. This is not an ...
Neil Slater's user avatar
  • 31.7k
1 vote

How to deal with the addition of a new state to the environment during training?

There are several ways to tackle this, although exploration is definitely not a solved problem yet ;) In general, I believe the right thing to do here is to measure the uncertainty of your policy or Q-...
harwiltz's user avatar
  • 1,126
1 vote
Accepted

Why can't we fully exploit the environment after the first episode in Q-learning?

You can't guarantee that you have taken every action from every state, even with 1000 time steps. There would be multiple outcomes: The episode terminates, either by success or failure before the ...
BlueTurtle's user avatar
1 vote
Accepted

Can tabular Q-learning converge even if it doesn't explore all state-action pairs?

In the tabular case, then the Q table will only converge if you have walked around the whole of the table. Note that to guarantee convergence we need $\sum\limits_{n=1}^{\infty}\alpha_n(a) = \infty$ ...
David's user avatar
  • 4,770

Only top scored, non community-wiki answers of a minimum length are eligible