# Questions tagged [epsilon-greedy-policy]

For questions about the $\epsilon$-greedy policy, which is typically used as a behavioural policy (i.e. a policy used to interact with the environment) during the interaction of reinforcement learning agents with the environment.

29 questions
Filter by
Sorted by
Tagged with
1 vote
21 views

### Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts

I am currently studying the equations 5.2 in Reinforcement Learning An Introduction By Sutton and Barto on page 101. I want to comprehent the proof by a simple example: Having only one State with two ...
• 13
1 vote
86 views

### Why are these two implementations of the $\epsilon$-greedy policy different?

According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:  \pi(a|s) = \begin{cases} \frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } ...
• 185
1 vote
262 views

### Why is my DQN agent not converging to a constant reward?

I'm currently training a DQN agent. I use an epsilon greedy exploration strategy where I decay the epsilon value linearly until it reaches 0 over 300 episodes. For the rest of the remaining 50 ...
69 views

### When do you use epsilon in Reinforcement Learning?

In reinforcement learning (DQN) do I use epsilon when I am collecting examples from the environment or do I use epsilon when I am training the Q network and Target network ?
• 95
117 views

### In DQN, how to increase epsilon but not too much?

I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to ...
91 views

### Epsilon-greedy action selection question

Is the following sentence correct? The estimated values $Q(a)$ do not converge to the true values $q_*(a)$ because $\epsilon$-greedy action selection behaves randomly from time to time. My Answer: The ...
1 vote
450 views

### If a policy is epsilon-greedy, is it technically stochastic?

Even though if exploration doesn't happen, it's deterministic.
• 93
274 views

### Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
• 33
1k views

### How is the probability of a greedy action in "$\epsilon$-greedy policies" derived?

In Sutton & Barto's book on reinforcement learning (section 5.4, p. 100) we have the following: The on-policy method we present in this section uses $\epsilon$ greedy policies, meaning that most ...
• 199
638 views

### How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?

I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy. Can someone tell me how to code an $\epsilon$-soft policy? I know how to ...
• 23
4k views

### What is the probability of selecting the greedy action in a 0.5-greedy selection method for the 2-armed bandit problem?

I'm new to reinforcement learning and I'm going through Sutton and Barto. Exercise 2.1 states the following: In $\varepsilon$-greedy action selection, for the case of two actions and $\varepsilon=0.5$...
• 575
1k views

### How to fight with unstability in self play?

I'm working on a neural network that plays some board games like reversi or tic-tac-toe (zero-sum games, two players). I'm trying to have one network topology for all the games - I specifically don't ...
• 141
1 vote
44 views

4k views

### What should the value of epsilon be in the Q-learning?

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles. What I don't understand is the impact of $\epsilon$. What value ...
• 353
53 views

### Off-policy full-random training in easy-to-explore environment

Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ...
• 266
1 vote
396 views

### Should the exploration rate be updated at the end of the episode or at every step?

My agent uses an $\epsilon$-greedy strategy to learn. The exploration rate (i.e. $\epsilon$) decays throughout the training. I've seen examples where people update $\epsilon$ every time an action is ...
• 773
255 views

### Can we stop training as soon as epsilon is small?

I'm new to reinforcement learning. As it is common in RL, $\epsilon$-greedy search for the behavior/exploration is used. So, at the beginning of the training, $\epsilon$ is high, and therefore a lot ...
1 vote
823 views

### Understanding GLIE conditions for epsilon greedy approach

I was going through this course on reinforcement learning (the course has two lecture videos and corresponding slides) and I had a doubt. On slide 18 of this pdf, it states following condition for an ...
• 151
158 views

### What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?

I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that ...
3k views

### What happens when you select actions using softmax instead of epsilon greedy in DQN?

I understand the two major branches of RL are Q-Learning and Policy Gradient methods. From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as ...
1 vote
3k views

### Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem? [closed]

I ran a test using 3 strategies for multi-armed bandit: UCB, $\epsilon$-greedy, and Thompson sampling. The results for the rewards I got are as follows: Thompson sampling had the highest average ...
415 views

### Is this proof of $\epsilon$-greedy policy improvement correct?

The following paragraph about $\epsilon$-greedy policies can be found at the end of page 100, under section 5.4, of the book "Reinforcement Learning: An Introduction" by Richard Sutton and ...
• 147
4k views

### Is there an advantage in decaying $\epsilon$ during Q-Learning?

If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence?
• 309
8k views

### What is the difference between the $\epsilon$-greedy and softmax policies?

Could someone explain to me which is the key difference between the $\epsilon$-greedy policy and the softmax policy? In particular, in the contest of SARSA and Q-Learning algorithms. I understood the ...
• 199
1 vote