Questions tagged [epsilon-greedy-policy]

For questions about the $\epsilon$-greedy policy, which is typically used as a behavioural policy (i.e. a policy used to interact with the environment) during the interaction of reinforcement learning agents with the environment.

Filter by
Sorted by
Tagged with
1 vote
1 answer
72 views

Why are these two implementations of the $\epsilon$-greedy policy different?

According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as: $$ \pi(a|s) = \begin{cases} \frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } ...
kklaw's user avatar
  • 175
1 vote
1 answer
151 views

Why is my DQN agent not converging to a constant reward?

I'm currently training a DQN agent. I use an epsilon greedy exploration strategy where I decay the epsilon value linearly until it reaches 0 over 300 episodes. For the rest of the remaining 50 ...
gondorian's user avatar
0 votes
1 answer
55 views

When do you use epsilon in Reinforcement Learning?

In reinforcement learning (DQN) do I use epsilon when I am collecting examples from the environment or do I use epsilon when I am training the Q network and Target network ?
Stef's user avatar
  • 95
0 votes
1 answer
70 views

In DQN, how to increase epsilon but not too much?

I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to ...
Mouad's user avatar
  • 3
0 votes
1 answer
85 views

Epsilon-greedy action selection question

Is the following sentence correct? The estimated values $Q(a)$ do not converge to the true values $q_*(a)$ because $\epsilon$-greedy action selection behaves randomly from time to time. My Answer: The ...
Gunners 's user avatar
1 vote
3 answers
390 views

If a policy is epsilon-greedy, is it technically stochastic?

Even though if exploration doesn't happen, it's deterministic.
Melanol's user avatar
  • 93
0 votes
1 answer
260 views

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
Aquila's user avatar
  • 33
3 votes
2 answers
860 views

How is the probability of a greedy action in "$\epsilon$-greedy policies" derived?

In Sutton & Barto's book on reinforcement learning (section 5.4, p. 100) we have the following: The on-policy method we present in this section uses $\epsilon$ greedy policies, meaning that most ...
user3489173's user avatar
2 votes
1 answer
579 views

How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?

I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy. Can someone tell me how to code an $\epsilon$-soft policy? I know how to ...
A Q's user avatar
  • 23
6 votes
1 answer
3k views

What is the probability of selecting the greedy action in a 0.5-greedy selection method for the 2-armed bandit problem?

I'm new to reinforcement learning and I'm going through Sutton and Barto. Exercise 2.1 states the following: In $\varepsilon$-greedy action selection, for the case of two actions and $\varepsilon=0.5$...
Daviiid's user avatar
  • 573
3 votes
2 answers
987 views

How to fight with unstability in self play?

I'm working on a neural network that plays some board games like reversi or tic-tac-toe (zero-sum games, two players). I'm trying to have one network topology for all the games - I specifically don't ...
Maras's user avatar
  • 141
1 vote
0 answers
44 views

$\epsilon$-greedy policy in environments where actions are performed in a long term. Does it has influence?

I'm working in an environment where once an action $a \in A$ is performed, it must hold this action selection for a while. To clarify this, assumes a horizon length $h$ and the set of actions: $\{a_{1}...
HenDoNR's user avatar
  • 81
5 votes
1 answer
803 views

Why does Q-learning converge under 100% exploration rate?

I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
Rim Sleimi's user avatar
5 votes
1 answer
795 views

Multi Armed Bandits with large number of arms

I'm dealing with a (stochastic) Multi Armed Bandit (MAB) with a large number of arms. Consider a pizza machine that produces a pizza depending on an input $i$ (equivalent to an arm). The (finite) set ...
D. B.'s user avatar
  • 101
1 vote
1 answer
139 views

Should my agent be taking varying number of steps?

My environment is set up so that my self-driving agent can take maximum of 400 steps (which is the end goal) before it resets with a completion reward. Despite attaining the end goal during the $\...
desert_ranger's user avatar
3 votes
1 answer
4k views

What should the value of epsilon be in the Q-learning?

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles. What I don't understand is the impact of $\epsilon$. What value ...
Exploring's user avatar
  • 321
0 votes
0 answers
53 views

Off-policy full-random training in easy-to-explore environment

Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ...
Loheek's user avatar
  • 266
1 vote
0 answers
364 views

Should the exploration rate be updated at the end of the episode or at every step?

My agent uses an $\epsilon$-greedy strategy to learn. The exploration rate (i.e. $\epsilon$) decays throughout the training. I've seen examples where people update $\epsilon$ every time an action is ...
mark mark's user avatar
  • 763
3 votes
1 answer
241 views

Can we stop training as soon as epsilon is small?

I'm new to reinforcement learning. As it is common in RL, $\epsilon$-greedy search for the behavior/exploration is used. So, at the beginning of the training, $\epsilon$ is high, and therefore a lot ...
Micha Christ's user avatar
1 vote
0 answers
738 views

Understanding GLIE conditions for epsilon greedy approach

I was going through this course on reinforcement learning (the course has two lecture videos and corresponding slides) and I had a doubt. On slide 18 of this pdf, it states following condition for an ...
Mahesha999's user avatar
4 votes
1 answer
153 views

What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?

I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that ...
Metrician's user avatar
7 votes
1 answer
3k views

What happens when you select actions using softmax instead of epsilon greedy in DQN?

I understand the two major branches of RL are Q-Learning and Policy Gradient methods. From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as ...
Linsu Han's user avatar
1 vote
1 answer
3k views

Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem? [closed]

I ran a test using 3 strategies for multi-armed bandit: UCB, $\epsilon$-greedy, and Thompson sampling. The results for the rewards I got are as follows: Thompson sampling had the highest average ...
Java coder's user avatar
6 votes
1 answer
381 views

Is this proof of $\epsilon$-greedy policy improvement correct?

The following paragraph about $\epsilon$-greedy policies can be found at the end of page 100, under section 5.4, of the book "Reinforcement Learning: An Introduction" by Richard Sutton and ...
Nishanth Rao's user avatar
3 votes
1 answer
4k views

Is there an advantage in decaying $\epsilon$ during Q-Learning?

If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence?
KaneM's user avatar
  • 309
3 votes
1 answer
7k views

What is the difference between the $\epsilon$-greedy and softmax policies?

Could someone explain to me which is the key difference between the $\epsilon$-greedy policy and the softmax policy? In particular, in the contest of SARSA and Q-Learning algorithms. I understood the ...
FraMan's user avatar
  • 199
1 vote
0 answers
593 views

Why is the $\epsilon$ hyper-parameter (in the $\epsilon$-greedy policy) annealed smoothly?

As far as I understand, RL is a process that can be divided into 2 stages: Exploring a wide range of paths (acting randomly) Refining the current optimal paths (revolving around actions with a so-...
Kari's user avatar
  • 270
2 votes
1 answer
321 views

Does eligibility traces and epsilon-greedy do the same task in different ways?

I understand that, in Reinforcement Learning algorithms, such as Q-learning, to prevent selecting the actions with greatest q-values too fast and allow for exploration, we use eligibility traces. Here ...
Abhishek Dhyani's user avatar