Questions tagged [exploration-strategies]

For questions about exploration strategies (or techniques) used in reinforcement learning or bandit problems. Examples of exploration strategies are random strategy, $\epsilon$-greedy, greedy (no exploration), Upper Confidence Bound (UCB), or Thompson sampling.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

thompson, UCB, e-greedy vs information state space algorithms for bandits

I am trying to understand why UCB, thompson sampling etc are inferior to information state space bandits in certain cases. Consider page 25 topic "Value of information" and below in the ...
hugh's user avatar
  • 23
0 votes
0 answers
94 views

Methods for sequential decision optimization problem with nonlinear bayesian reward function

I am attempting to grasp if there are any other methods out there that i am not aware of that can be beneficial given my problem context. Being inspired from optimal experimental design communities ...
paul's user avatar
  • 33
0 votes
1 answer
45 views

In DQN, how to increase epsilon but not too much?

I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to ...
Mouad's user avatar
  • 3
1 vote
1 answer
56 views

Metrics to compare the exploraation of RL Algorithms

I am looking for metrics to compare the exploration under different RL Algos/reward functions. I want to somehow quantify how big of a region of the policy space is explored. What are common measures ...
Lukas Schroth's user avatar
2 votes
0 answers
217 views

Choosing and Designing Decay Types for Epsilon-Greedy Exploration in Reinforcement Learning

I am working on a reinforcement learning project that involves epsilon-greedy exploration. I have two questions regarding the choice between linear and exponential decay for epsilon, and the ...
XiaoBanni's user avatar
1 vote
0 answers
47 views

Can entropy bonus be used with state-independent log std for stochastic policies?

In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
flxh's user avatar
  • 131
0 votes
1 answer
235 views

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
Aquila's user avatar
  • 33
1 vote
0 answers
51 views

Alternatives to using RL to explore an environment

I am looking at some ideas on exploring an environment using Curiosity Driven Exploration and am being a little skeptical about it. The objective here is to just explore without the need to obtain ...
desert_ranger's user avatar
1 vote
1 answer
284 views

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

Consider the following algorithm from the textbook titled Reinforcement Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart While playing the game for the generation of ...
hanugm's user avatar
  • 3,581
1 vote
1 answer
112 views

Are the two policies in SARSA for choosing an action the same?

Here is the pseudocode for SARSA (which I took from here) Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, ...
PeterBe's user avatar
  • 212
3 votes
1 answer
145 views

Is it possible to apply a particular exploration policy for the on-policy RL agents?

Is it possible to use any particular strategy to explore (e.g. metaheuristics) in on-policy algorithms (e.g. in PPO) or is it only possible to define particular policies to explore in off-policy ...
Pulse9's user avatar
  • 282
0 votes
1 answer
241 views

How to reduce the number of episodes before the agent learns in this game?

The initial environment state is 0.25. Each time step the agent performs a discrete action of 0 or ...
penkovsky's user avatar
  • 264
1 vote
1 answer
70 views

Which policy has to be followed by a player while construction of its own Q-table?

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
satya's user avatar
  • 187
0 votes
1 answer
300 views

Which policy do I need to use in updating Q function?

Policy function can be of two types: deterministic policy and stochastic policy. Deterministic policy is of the form $\pi : S \rightarrow A$ Stochastic policy is defined using conditional probability ...
satya's user avatar
  • 187
1 vote
0 answers
419 views

(explore-exploit + supervised learning ) vs contextual bandits

Lets take an ad recommendation problem for 1 slot. Feedback is click/no click. I can solve this by contextual bandits. But I can also introduce exploration in supervised learning, I learn my model ...
dksahuji's user avatar
  • 111
1 vote
1 answer
89 views

In addition to the reward function, which other functions do I need to implement Q-learning?

In general, $Q$ function is defined as $$Q : S \times A \rightarrow \mathbb{R}$$ $$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$ $\alpha$ and $\gamma$...
satya's user avatar
  • 187
5 votes
1 answer
654 views

Why does Q-learning converge under 100% exploration rate?

I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
Rim Sleimi's user avatar
5 votes
3 answers
650 views

In Q-learning, wouldn't it be better to simply iterate through all possible states?

In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state ...
Kricket's user avatar
  • 197
1 vote
0 answers
433 views

Are actions deterministic during testing in continuous action space PPO?

In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent ...
Mika's user avatar
  • 331
3 votes
1 answer
377 views

Strategy for playing a board game with Minimax algorithm

I want to build a player for the following game: You have a board where position 1 is your player, position 2 is the rival ...
vesii's user avatar
  • 81
3 votes
3 answers
534 views

Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

Why aren't exploration techniques, such as UCB or Thompson sampling, typically used in bandit problems, used in full RL problems? Monte Carlo Tree Search may use the above-mentioned methods in its ...
Mika's user avatar
  • 331
2 votes
1 answer
305 views

In DQN, is it possible to make some actions more likely?

In a general DQN framework, if I have an idea of some actions being better than some other actions, is it possible to make the agent select the better actions more often?
user3656142's user avatar
3 votes
1 answer
3k views

Is there an advantage in decaying $\epsilon$ during Q-Learning?

If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence?
KaneM's user avatar
  • 309
6 votes
1 answer
507 views

What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?

In the context of reinforcement learning, a policy, $\pi$, is often defined as a function from the space of states, $\mathcal{S}$, to the space of actions, $\mathcal{A}$, that is, $\pi : \mathcal{S} \...
nbro's user avatar
  • 39.1k
6 votes
1 answer
2k views

Should I be decaying the learning rate and the exploration rate in the same manner?

Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
rtz's user avatar
  • 91