2
$\begingroup$

I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy.

Can someone tell me how to code an $\epsilon$-soft policy?

I know how to code the $\epsilon$-greedy. In $\epsilon$-soft, there are inequalities in place of equalities which is the issue for coding the $\epsilon$-soft.

$\endgroup$
1
$\begingroup$

You cannot code an $\epsilon$-soft policy directly, because it is not specific enough.

A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for choosing any action, where $\mathcal{A}$ is the set of all possible actions.

I know how to code the $\epsilon$-greedy.

Then you already know how to code the most commonly-used $\epsilon$-soft policy, because an $\epsilon$-greedy policy is an $\epsilon$-soft policy.

there are inequalities in place of equalities which is the issue for coding the $\epsilon$-soft

That is correct. In fact $\epsilon$-soft can be thought of as a constraint or test. So you could write some code that tested whether any policy was an $\epsilon$-soft policy for any given value of $\epsilon$. Or you could write code that determined the value of $\epsilon$ for any policy.

Slightly harder would be code that forced a supplied policy to meet constraints of being $\epsilon$-soft, because adjusting any low probabilities to be high enough would mean reducing other probabilities, and there are multiple ways you could do that.

However, a really simple way to make any starting policy $\pi$ into an $\epsilon$-soft variant is to make the policy choice in 2 steps - first step choose between the original policy with probability $(1-\epsilon)$, and with probability $\epsilon$ choose a fixed policy with equal probability for each action. Second step, evaluate whichever policy the first step chose to determine the action.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.