Questions tagged [exploration-exploitation-tradeoff]

For questions related to the exploration-exploitation trade-off (or dilemma) in the context of reinforcement learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
21 views

thompson, UCB, e-greedy vs information state space algorithms for bandits

I am trying to understand why UCB, thompson sampling etc are inferior to information state space bandits in certain cases. Consider page 25 topic "Value of information" and below in the ...
hugh's user avatar
  • 23
1 vote
1 answer
23 views

Practical differences between the different RL architectures

I have tried many different RL architectures: DQN, PPO, Policy optimization, and for my specific problem they all failed in their basic setup. Eventually I discovered that my problem had too sparse/...
Erik Storm's user avatar
0 votes
0 answers
94 views

Methods for sequential decision optimization problem with nonlinear bayesian reward function

I am attempting to grasp if there are any other methods out there that i am not aware of that can be beneficial given my problem context. Being inspired from optimal experimental design communities ...
paul's user avatar
  • 33
2 votes
1 answer
167 views

In which community does using a Bayesian regression model as a reward function with exploration vs. exploitation challenges fall under?

I am trying to find research papers addressing a problem that, in my opinion, deserves significant attention. However, I am having difficulty locating relevant information. To illustrate the problem ...
paul's user avatar
  • 33
0 votes
1 answer
45 views

In DQN, how to increase epsilon but not too much?

I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to ...
Mouad's user avatar
  • 3
1 vote
1 answer
56 views

Metrics to compare the exploraation of RL Algorithms

I am looking for metrics to compare the exploration under different RL Algos/reward functions. I want to somehow quantify how big of a region of the policy space is explored. What are common measures ...
Lukas Schroth's user avatar
2 votes
0 answers
218 views

Choosing and Designing Decay Types for Epsilon-Greedy Exploration in Reinforcement Learning

I am working on a reinforcement learning project that involves epsilon-greedy exploration. I have two questions regarding the choice between linear and exponential decay for epsilon, and the ...
XiaoBanni's user avatar
1 vote
2 answers
57 views

Are RL algorithms suppose to keep learning?

I don't understand if the purposes of RL agents is simply optimizing a model with a reward instead of using labeled data (i.e. in a supervision fashion), or they have also the purpose of keep training ...
pippo's user avatar
  • 41
4 votes
1 answer
954 views

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
Vladimir Belik's user avatar
5 votes
6 answers
3k views

Why is exploitation necessary during training?

I have read many blog articles making all kinds of broad analogies to explain the exploration/exploitation trade-off. However, I still can't fully grasp it. On an extremely abstract level, I ...
Vladimir Belik's user avatar
0 votes
1 answer
241 views

How to reduce the number of episodes before the agent learns in this game?

The initial environment state is 0.25. Each time step the agent performs a discrete action of 0 or ...
penkovsky's user avatar
  • 264
1 vote
1 answer
123 views

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
user529295's user avatar
1 vote
1 answer
70 views

Which policy has to be followed by a player while construction of its own Q-table?

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
satya's user avatar
  • 187
5 votes
3 answers
651 views

In Q-learning, wouldn't it be better to simply iterate through all possible states?

In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state ...
Kricket's user avatar
  • 197
2 votes
0 answers
362 views

Why is the ideal exploration parameter in the UCT algorithm $\sqrt{2}$?

From Wikipedia, in the Monte-Carlo Tree Search algorithm, you should choose the node that maximizes the value: $${\displaystyle {\frac {w_{i}}{n_{i}}}+c{\sqrt {\frac {\ln N_{i}}{n_{i}}}}},$$ where ${...
Gilad Felsen's user avatar
1 vote
0 answers
268 views

What is the difference between exploitation and exploration in the context of optimization?

In the paper Moth-flame optimization algorithm: A novel nature-inspired heuristic paradigm (2015, published in Knowledge-Based Systems) The test functions are divided to three groups: unimodal, multi-...
user2752471's user avatar
1 vote
0 answers
314 views

Should the exploration rate be updated at the end of the episode or at every step?

My agent uses an $\epsilon$-greedy strategy to learn. The exploration rate (i.e. $\epsilon$) decays throughout the training. I've seen examples where people update $\epsilon$ every time an action is ...
mark mark's user avatar
  • 753
3 votes
1 answer
212 views

Can we stop training as soon as epsilon is small?

I'm new to reinforcement learning. As it is common in RL, $\epsilon$-greedy search for the behavior/exploration is used. So, at the beginning of the training, $\epsilon$ is high, and therefore a lot ...
Micha Christ's user avatar
4 votes
2 answers
996 views

What is the meaning of "exploration" in reinforcement and supervised learning?

While exploration is an integral part of reinforcement learning (RL), it does not pertain to supervised learning (SL) since the latter is already provided with the data set from the start. That said, ...
Tfovid's user avatar
  • 177
2 votes
1 answer
173 views

What is the optimal exploration-exploitation trade-off in Q*bert?

I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but ...
Ryan Rudes's user avatar
0 votes
0 answers
81 views

How to modify the Actor-Critic policy gradient algorithm to perform Safe exploration in Reinforcement Learning

I am trying to implement safe exploration technique in [Ref.1]. I am using Soft Actor-Critic algorithm to teach an agent to introduce a bias between 0 and 1 to a specific state of interest in my ...
Ayomi Al-noor's user avatar
2 votes
1 answer
167 views

Why do some DQN implementations not require random exploration but instead emulate all actions?

I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all ...
unter_983's user avatar
  • 331
2 votes
1 answer
498 views

Why is 100% exploration bad during the learning stage in reinforcement learning?

Why can't we during the first 1000 episodes allow our agent to perform only exploration? This will give a better chance of covering the entire space state. Then, after the number of episodes, we can ...
Chukwudi's user avatar
  • 349
3 votes
2 answers
209 views

Why is it not advisable to have a 100 percent exploration rate? [duplicate]

During the learning phase, why don't we have a 100% exploration rate, to allow our agent to fully explore our environment and update the Q values, then during testing we bring in exploitation? Does ...
Chukwudi's user avatar
  • 349
1 vote
1 answer
71 views

How to deal with the addition of a new state to the environment during training?

Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state. When it gets to that new ...
Chukwudi's user avatar
  • 349
1 vote
2 answers
62 views

Why do we explore after we have an accurate estimate of the value function?

Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small ...
Chukwudi's user avatar
  • 349
1 vote
2 answers
145 views

Why can't we fully exploit the environment after the first episode in Q-learning?

During the first episode, it's 100% exploration, because all our Q values are 0. Suppose we have 1000 time steps, and it's terminated by meeting a reward. So, after the first episode, why can't we ...
Chukwudi's user avatar
  • 349
4 votes
2 answers
2k views

Should I use exploration strategy in Policy Gradient algorithms?

In policy gradient algorithms the output is a stochastic policy - a probability for each action. I believe that if I follow the policy (sample an action from the policy) I make use of exploration ...
gnikol's user avatar
  • 175
1 vote
1 answer
488 views

Can tabular Q-learning converge even if it doesn't explore all state-action pairs?

My understanding of tabular Q-learning is that it essentially builds a dictionary of state-action pairs, so as to maximize the Markovian (i.e., step-wise, history-agnostic?) reward. This incremental ...
Tfovid's user avatar
  • 177
1 vote
0 answers
270 views

How can I increase the exploration in the Proximal Policy Optimation algorithm?

How can I increase the exploration in the Proximal Policy Optimation reinforcement learning algorithm? Is there a variable assigned for this purpose? I'm using the stable-baseline implementation: ...
Saeid Ghafouri's user avatar
1 vote
1 answer
71 views

Should I just use exploitation after I have trained the Q agent?

When using a trained Q-learning algorithm in an actual game, would I just use exploitation and no longer use exploration? Should I use exploration only during the training phase?
mason7663's user avatar
  • 603
1 vote
0 answers
588 views

Why is the $\epsilon$ hyper-parameter (in the $\epsilon$-greedy policy) annealed smoothly?

As far as I understand, RL is a process that can be divided into 2 stages: Exploring a wide range of paths (acting randomly) Refining the current optimal paths (revolving around actions with a so-...
Kari's user avatar
  • 250
8 votes
1 answer
680 views

What is the purpose of the actor in actor-critic algorithms?

For discrete action spaces, what is the purpose of the actor in actor-critic algorithms? My current understanding is that the critic estimates the future reward given an action, so why not just take ...
David Rein's user avatar
2 votes
1 answer
315 views

Does eligibility traces and epsilon-greedy do the same task in different ways?

I understand that, in Reinforcement Learning algorithms, such as Q-learning, to prevent selecting the actions with greatest q-values too fast and allow for exploration, we use eligibility traces. Here ...
Abhishek Dhyani's user avatar