Questions tagged [epsilon-greedy-policy]
For questions about the $\epsilon$-greedy policy, which is typically used as a behavioural policy (i.e. a policy used to interact with the environment) during the interaction of reinforcement learning agents with the environment.
27
questions
1
vote
1
answer
60
views
Why is my DQN agent not converging to a constant reward?
I'm currently training a DQN agent. I use an epsilon greedy exploration strategy where I decay the epsilon value linearly until it reaches 0 over 300 episodes. For the rest of the remaining 50 ...
- 35
0
votes
1
answer
33
views
When do you use epsilon in Reinforcement Learning?
In reinforcement learning (DQN) do I use epsilon when I am collecting examples from the environment or do I use epsilon when I am training the Q network and Target network ?
- 53
0
votes
1
answer
45
views
In DQN, how to increase epsilon but not too much?
I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to ...
- 3
0
votes
1
answer
73
views
Epsilon-greedy action selection question
Is the following sentence correct?
The estimated values $Q(a)$ do not converge to the true values $q_*(a)$ because $\epsilon$-greedy action selection behaves randomly from time to time.
My Answer: The ...
- 1
1
vote
3
answers
326
views
If a policy is epsilon-greedy, is it technically stochastic?
Even though if exploration doesn't happen, it's deterministic.
- 93
0
votes
1
answer
236
views
Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?
In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
- 33
3
votes
2
answers
634
views
How is the probability of a greedy action in "$\epsilon$-greedy policies" derived?
In Sutton & Barto's book on reinforcement learning (section 5.4, p. 100) we have the following:
The on-policy method we present in this section uses $\epsilon$ greedy policies, meaning that most ...
- 179
2
votes
1
answer
455
views
How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?
I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy.
Can someone tell me how to code an $\epsilon$-soft policy?
I know how to ...
- 23
6
votes
1
answer
3k
views
What is the probability of selecting the greedy action in a 0.5-greedy selection method for the 2-armed bandit problem?
I'm new to reinforcement learning and I'm going through Sutton and Barto. Exercise 2.1 states the following:
In $\varepsilon$-greedy action selection, for the case of two actions and $\varepsilon=0.5$...
- 563
3
votes
2
answers
914
views
How to fight with unstability in self play?
I'm working on a neural network that plays some board games like reversi or tic-tac-toe (zero-sum games, two players). I'm trying to have one network topology for all the games - I specifically don't ...
- 141
1
vote
0
answers
42
views
$\epsilon$-greedy policy in environments where actions are performed in a long term. Does it has influence?
I'm working in an environment where once an action $a \in A$ is performed, it must hold this action selection for a while. To clarify this, assumes a horizon length $h$ and the set of actions: $\{a_{1}...
- 81
5
votes
1
answer
655
views
Why does Q-learning converge under 100% exploration rate?
I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
- 215
5
votes
1
answer
707
views
Multi Armed Bandits with large number of arms
I'm dealing with a (stochastic) Multi Armed Bandit (MAB) with a large number of arms.
Consider a pizza machine that produces a pizza depending on an input $i$ (equivalent to an arm). The (finite) set ...
- 101
1
vote
1
answer
123
views
Should my agent be taking varying number of steps?
My environment is set up so that my self-driving agent can take maximum of 400 steps (which is the end goal) before it resets with a completion reward. Despite attaining the end goal during the $\...
- 588
2
votes
1
answer
3k
views
What should the value of epsilon be in the Q-learning?
I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles.
What I don't understand is the impact of $\epsilon$. What value ...
- 253
0
votes
0
answers
53
views
Off-policy full-random training in easy-to-explore environment
Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe).
In those environments, using off-policy algorithm, is it a good ...
- 266
1
vote
0
answers
314
views
Should the exploration rate be updated at the end of the episode or at every step?
My agent uses an $\epsilon$-greedy strategy to learn. The exploration rate (i.e. $\epsilon$) decays throughout the training. I've seen examples where people update $\epsilon$ every time an action is ...
- 753
3
votes
1
answer
212
views
Can we stop training as soon as epsilon is small?
I'm new to reinforcement learning.
As it is common in RL, $\epsilon$-greedy search for the behavior/exploration is used. So, at the beginning of the training, $\epsilon$ is high, and therefore a lot ...
- 31
1
vote
0
answers
640
views
Understanding GLIE conditions for epsilon greedy approach
I was going through this course on reinforcement learning (the course has two lecture videos and corresponding slides) and I had a doubt. On slide 18 of this pdf, it states following condition for an ...
- 141
4
votes
1
answer
148
views
What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?
I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that ...
- 95
7
votes
1
answer
2k
views
What happens when you select actions using softmax instead of epsilon greedy in DQN?
I understand the two major branches of RL are Q-Learning and Policy Gradient methods.
From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as ...
- 73
1
vote
1
answer
3k
views
Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem? [closed]
I ran a test using 3 strategies for multi-armed bandit: UCB, $\epsilon$-greedy, and Thompson sampling.
The results for the rewards I got are as follows:
Thompson sampling had the highest average ...
- 11
6
votes
1
answer
355
views
Is this proof of $\epsilon$-greedy policy improvement correct?
The following paragraph about $\epsilon$-greedy policies can be found at the end of page 100, under section 5.4, of the book "Reinforcement Learning: An Introduction" by Richard Sutton and ...
- 147
3
votes
1
answer
3k
views
Is there an advantage in decaying $\epsilon$ during Q-Learning?
If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence?
- 309
3
votes
1
answer
7k
views
What is the difference between the $\epsilon$-greedy and softmax policies?
Could someone explain to me which is the key difference between the $\epsilon$-greedy policy and the softmax policy? In particular, in the contest of SARSA and Q-Learning algorithms. I understood the ...
- 189
1
vote
0
answers
588
views
Why is the $\epsilon$ hyper-parameter (in the $\epsilon$-greedy policy) annealed smoothly?
As far as I understand, RL is a process that can be divided into 2 stages:
Exploring a wide range of paths (acting randomly)
Refining the current optimal paths (revolving around actions with a so-...
- 250
2
votes
1
answer
315
views
Does eligibility traces and epsilon-greedy do the same task in different ways?
I understand that, in Reinforcement Learning algorithms, such as Q-learning, to prevent selecting the actions with greatest q-values too fast and allow for exploration, we use eligibility traces.
Here ...
