Skip to main content
Share Your Experience: Take the 2024 Developer Survey
7 votes

What is the difference between the $\epsilon$-greedy and softmax policies?

The $\epsilon$-greedy policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability $1-\epsilon \in [0, 1]$ and a random action with probability $...
nbro's user avatar
  • 40.8k
6 votes
Accepted

Is this proof of $\epsilon$-greedy policy improvement correct?

The weights do sum to one. Note that in the second line where we have $$\frac{\epsilon}{|\mathcal{A}(s)|} \sum_a q_{\pi}(s,a) + (1-\epsilon)\max_aq_{\pi}(s,a) \; ,$$ the sum is over the whole action ...
David's user avatar
  • 4,920
6 votes
Accepted

What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?

This expression: $|\mathcal{A}(s)|$ means $|\quad|$ the size of $\mathcal{A}(s)$ the set of actions in state $s$ or more simply the number of actions allowed in the state. This makes sense in the ...
Neil Slater's user avatar
  • 32.5k
5 votes
Accepted

Why does Q-learning converge under 100% exploration rate?

Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random ...
David's user avatar
  • 4,920
5 votes
Accepted

What is the probability of selecting the greedy action in a 0.5-greedy selection method for the 2-armed bandit problem?

I read section 2.2 of Sutton and Barto, and I understand your confusion: the $\epsilon$-greedy algorithm is not defined precisely on page 27-28. Selecting an action randomly "every once in awhile&...
DeepQZero's user avatar
  • 1,399
4 votes
Accepted

What happens when you select actions using softmax instead of epsilon greedy in DQN?

DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action. This is a very common choice, because it is simple to implement and quite robust....
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

Is there an advantage in decaying $\epsilon$ during Q-Learning?

Yes Q-learning benefits from decaying epsilon in at least two ways: Early exploration. It makes little sense to follow whatever policy is implied by the initialised network closely, and more will be ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

How to fight with unstability in self play?

The AlphaZero paper mentions an "evaluation" step that seems to deal with the the problem similar to yours: ... we evaluate each new neural network checkpoint against the current best ...
Kostya's user avatar
  • 2,534
3 votes

If a policy is epsilon-greedy, is it technically stochastic?

I would argue it is just stochastic because it chooses the current best action with probability $1-\epsilon+\epsilon/|A|$ and then selects randomly among the rest of the actions with the remaining ...
sma's user avatar
  • 823
3 votes

How to fight with unstability in self play?

When in an environment with competing agents, from the perspective of each agent, the environment becomes non-markovian. That occurs because each agent is constantly adapting its own strategy to other'...
Felipe Costa's user avatar
3 votes
Accepted

How is the probability of a greedy action in "$\epsilon$-greedy policies" derived?

I did not understand how the probability $1-\epsilon+\frac{\epsilon}{|\mathcal{A}|}$ It is the sum of two mutally-exclusive possibilities: The agent chooses to exploit, selecting the greedy action ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

Does eligibility traces and epsilon-greedy do the same task in different ways?

Epsilon-greedy is one method of making an agent explore the state space to ensure that the agent doesn't settle on a sub-optimal policy. By taking random actions, even with a small probability, the ...
Jaden Travnik's user avatar
3 votes

Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem?

The first thing to note here is that your results seem aligned with the results commonly found in the bandit literature. Second thing to note would be that the performance of bandit algorithms is ...
user5093249's user avatar
2 votes

What should the value of epsilon be in the Q-learning?

What does it mean when ϵ=0 and ϵ=1? If ϵ=1, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set ϵ=0, does ...
chessprogrammer's user avatar
2 votes

Can we stop training as soon as epsilon is small?

How much the $Q$-values change does not depend on the value of $\epsilon$, rather the value of $\epsilon$ dictates how likely you are to take a random action and thus take an action that could give ...
David's user avatar
  • 4,920
2 votes

How is the probability of a greedy action in "$\epsilon$-greedy policies" derived?

I did not understand how the probability 1−ϵ+ϵ/|A| It's about mutually exclusive probabilities. If A and B are mutually exclusive events, and C can happen in both: Then, P(C) = P(C$\cap$A) + P(C$\...
Melanol's user avatar
  • 93
2 votes

If a policy is epsilon-greedy, is it technically stochastic?

Yes - you can think of an epsilon-greedy policy as a mixture of a policy that chooses an action at random (the stochastic part) and a possibly deterministic policy used otherwise. The value of epsilon ...
mikkola's user avatar
  • 579
1 vote

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In single-step Q learning, you can use almost any exploration policy that you like, provided it covers all choices eventually. Usually you want to focus around the target policy, because that is the ...
Neil Slater's user avatar
  • 32.5k
1 vote
Accepted

Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts

Your highlighted equation 2 is incorporating the probabilities under policy $\pi'$ which as stated in your referenced page is the $\epsilon$-greedy policy with respect to $q_{\pi}$, the coefficients ($...
cinch's user avatar
  • 2,277
1 vote

Why are these two implementations of the $\epsilon$-greedy policy different?

The two implementations you posted are different, but they do represent the same $\epsilon$-greedy policy. The first function returns an array A which contains the ...
Neil Slater's user avatar
  • 32.5k
1 vote
Accepted

Why is my DQN agent not converging to a constant reward?

Annealing $\epsilon$ to 0 in $\epsilon$-greedy DQN is intended to reduce the exploration capability of the DQN agent, but it does not prevent the DQN agent from continuing to learn. Typically, DQN ...
DeepQZero's user avatar
  • 1,399
1 vote
Accepted

When do you use epsilon in Reinforcement Learning?

As you asked specifically about DQN, I will talk about Q-Learning. Since it is an off-policy algorithm, we can collect data with any exploratory policy $\beta$ whilst learning about the greedy policy $...
David's user avatar
  • 4,920
1 vote

Epsilon-greedy action selection question

I believe, that you are correct in saying the sentence is incorrect. Since, as you mention, the epsilon greedy action selection will just allow for the Q values to converge to the optimal q* value ...
chs's user avatar
  • 11
1 vote
Accepted

How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?

You cannot code an $\epsilon$-soft policy directly, because it is not specific enough. A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for ...
Neil Slater's user avatar
  • 32.5k
1 vote

Multi Armed Bandits with large number of arms

Without any knowledge on the references you came across, I am assuming that the authors were considering common applications of MAB (planning, online learning, etc.) for which the time horizon is ...
rhdxor's user avatar
  • 206
1 vote
Accepted

Should my agent be taking varying number of steps?

Your graph looks to me like a typical learning curve plotted for training process in reinforcement learning. Looking at it in detail I can say: There is clearly some learning occurring. There is a ...
Neil Slater's user avatar
  • 32.5k

Only top scored, non community-wiki answers of a minimum length are eligible