Questions tagged [exploration-strategies]
For questions about exploration strategies (or techniques) used in reinforcement learning or bandit problems. Examples of exploration strategies are random strategy, $\epsilon$-greedy, greedy (no exploration), Upper Confidence Bound (UCB), or Thompson sampling.
18
questions
0
votes
0
answers
12
views
Are temperature schedules used in Boltzmann Q-learning?
In Q-learning with the epsilon greedy method, I have seen implementations for training an agent that provide the possibility to pass epsilon schedules. Basically you can keep the epsilon constant or ...
- 101
1
vote
0
answers
45
views
Alternatives to using RL to explore an environment
I am looking at some ideas on exploring an environment using Curiosity Driven Exploration and am being a little skeptical about it. The objective here is to just explore without the need to obtain ...
- 235
1
vote
1
answer
55
views
Is it the high probability action that is always selected by the agent in REINFORCE algorithm?
Consider the following algorithm from the textbook titled Reinforcement
Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart
While playing the game for the generation of ...
- 2,975
1
vote
1
answer
55
views
Are the two policies in SARSA for choosing an action the same?
Here is the pseudocode for SARSA (which I took from here)
Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, ...
- 213
3
votes
1
answer
51
views
Is it possible to apply a particular exploration policy for the on-policy RL agents?
Is it possible to use any particular strategy to explore (e.g. metaheuristics) in on-policy algorithms (e.g. in PPO) or is it only possible to define particular policies to explore in off-policy ...
- 282
1
vote
1
answer
62
views
Which policy has to be followed by a player while construction of its own Q-table?
Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
- 187
0
votes
1
answer
110
views
Which policy do I need to use in updating Q function?
Policy function can be of two types: deterministic policy and stochastic policy.
Deterministic policy is of the form $\pi : S \rightarrow A$
Stochastic policy is defined using conditional probability ...
- 187
0
votes
0
answers
144
views
(explore-exploit + supervised learning ) vs contextual bandits
Lets take an ad recommendation problem for 1 slot. Feedback is click/no click. I can solve this by contextual bandits. But I can also introduce exploration in supervised learning, I learn my model ...
- 101
1
vote
1
answer
70
views
In addition to the reward function, which other functions do I need to implement Q-learning?
In general, $Q$ function is defined as
$$Q : S \times A \rightarrow \mathbb{R}$$
$$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$
$\alpha$ and $\gamma$...
- 187
5
votes
1
answer
288
views
Why does Q-learning converge under 100% exploration rate?
I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
- 215
5
votes
3
answers
523
views
In Q-learning, wouldn't it be better to simply iterate through all possible states?
In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state ...
- 197
1
vote
0
answers
140
views
Are actions deterministic during testing in continuous action space PPO?
In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent ...
- 167
3
votes
1
answer
231
views
Strategy for playing a board game with Minimax algorithm
I want to build a player for the following game:
You have a board where position 1 is your player, position 2 is the rival ...
- 81
3
votes
3
answers
128
views
Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?
Why aren't exploration techniques, such as UCB or Thompson sampling, typically used in bandit problems, used in full RL problems?
Monte Carlo Tree Search may use the above-mentioned methods in its ...
- 167
2
votes
1
answer
105
views
In DQN, is it possible to make some actions more likely?
In a general DQN framework, if I have an idea of some actions being better than some other actions, is it possible to make the agent select the better actions more often?
- 175
3
votes
1
answer
1k
views
Is there an advantage in decaying $\epsilon$ during Q-Learning?
If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence?
- 309
6
votes
1
answer
308
views
What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?
In the context of reinforcement learning, a policy, $\pi$, is often defined as a function from the space of states, $\mathcal{S}$, to the space of actions, $\mathcal{A}$, that is, $\pi : \mathcal{S} \...
- 33k
6
votes
1
answer
2k
views
Should I be decaying the learning rate and the exploration rate in the same manner?
Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
- 91