3
$\begingroup$

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles.

What I don't understand is the impact of $\epsilon$. What value should it take? $0.5$, $0.6$, or $0.7$?

What does it mean when $\epsilon = 0$ and $\epsilon = 1$? If $\epsilon = 1$, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set $\epsilon = 0$, does this imply that the agent doesn't explore?

For a typical problem, what is the recommended value for this parameter?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

What does it mean when ϵ=0 and ϵ=1? If ϵ=1, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set ϵ=0, does this imply that the agent doesn't explore?

You are correct, when ϵ=1 the agent acts randomly. When ϵ=0, the agent always takes the current greedy actions. Both of these scenarios are not ideal. Always acting greedily will prevent the agent from exploring possibly better parts of the state space, and instead the agent may get stuck in a local optimum. And always exploring randomly is obviously not ideal as well. Thus, we need to balance between these two. This is often called the balance between exploration and exploitation.

For a typical problem, what is the recommended value for this parameter?

ϵ is a hyper parameter. It is impossible to know in advance what the ideal value is, and it is highly dependent on the problem at hand. There is no general answer to this question.

That being said, the most common values that I have seen typically range between 0.01 and 0.1. But I want to stress, there is no ideal value that works for all problems. A typical strategy is to try several values and see which one works best. For more information, you might want to look up hyper parameter tuning.

Another common practice is slowly decaying epsilon over time (often this is called "annealing" or "simulated annealing"). Depending on the algorithm, decaying epsilon to zero may be a requirement for convergence. In some contexts, an algorithm that decays epsilon over time is called a GLIE algorithm. For example, see this.

Improve this answer
$\endgroup$
4
  • $\begingroup$ You say "agent explores randomly.", but this is misleading because "to explore" in RL already implies (usually) that it's randomly, i.e. you take a random action. So, I think that this answer can be made more precise. By the way, you can type epsilon in latex here as $\epsilon$. Moreover, there may be cases/problems where $\epsilon$ needs to be higher or smaller. To be honest, right now, no such an example comes to my mind, but it's likely there are some guidelines. It may also be worthing mentioning schedules to decay the this parameter. $\endgroup$
    – nbro
    Nov 26, 2020 at 13:04
  • $\begingroup$ @nbro. I will update answer. Regarding your final point, one example that comes to mind is in certain on policy algorithms, epsilon must he decayed to guarantee convergence. Sutton calls this GLIE. $\endgroup$
    – chessprogrammer
    Nov 26, 2020 at 13:49
  • $\begingroup$ Regarding guidelines for a higher epsilon: it is easy to construct a toy MDP where a higher epsilon converges faster. One classic example is a K armed bandit with very high variance. But its hard to draw a general guideline as to when it should be higher. $\endgroup$
    – chessprogrammer
    Nov 26, 2020 at 13:54
  • $\begingroup$ I wouldn't call annealing of the epsilon "simulated annealing" as "SA" refers to a specific algorithm/meta-heuristic, though it's possible that this term is also used in the context of annealing the epsilon. I don't remember that though. $\endgroup$
    – nbro
    Nov 26, 2020 at 19:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .