0
$\begingroup$

Sutton-Barto, page 102:

In the second paragraph, we have:

Consider a new environment that is just like the original environment, except with the requirement that policies be $\epsilon$-soft “moved inside” the environment."

Later in this paragraph, it is written as:

The best one can do in this new environment with general policies is the same as the best one could do in the original environment with $\epsilon$-soft policies.

Is there an inconsistency between the two statements because if we are allowed general policies in the new environment why do we need the requirement of $\epsilon$-soft policies “moved inside” the new environment?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

He's trying to turn the requirement "this policy is $\epsilon$-soft" into a feature of the environment. That way every policy in the new environment could be turned into an $\epsilon$-soft in the old.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But if we are allowed general policies in the new environment why do we need this requirement because general policies include $epsilon$-soft policies? $\endgroup$
    – DSPinfinity
    Commented Mar 25 at 23:37
  • $\begingroup$ The point of $\epsilon$-soft policies is to encourage exploration (important for finding good policies; I think it can be shown that training an RL agent w/ $\epsilon$-soft policies will be exponentially more likely to converge to the highest-reward policy). The point of making a new environment is so he doesn't need to put this requirement on the policy anymore. $\endgroup$
    – programjames
    Commented Mar 26 at 3:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .