How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?

Question

I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy.

Can someone tell me how to code an $\epsilon$-soft policy?

I know how to code the $\epsilon$-greedy. In $\epsilon$-soft, there are inequalities in place of equalities which is the issue for coding the $\epsilon$-soft.

Answer 1

You cannot code an $\epsilon$-soft policy directly, because it is not specific enough.

A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for choosing any action, where $\mathcal{A}$ is the set of all possible actions.

I know how to code the $\epsilon$-greedy.

Then you already know how to code the most commonly-used $\epsilon$-soft policy, because an $\epsilon$-greedy policy is an $\epsilon$-soft policy.

there are inequalities in place of equalities which is the issue for coding the $\epsilon$-soft

That is correct. In fact $\epsilon$-soft can be thought of as a constraint or test. So you could write some code that tested whether any policy was an $\epsilon$-soft policy for any given value of $\epsilon$. Or you could write code that determined the value of $\epsilon$ for any policy.

Slightly harder would be code that forced a supplied policy to meet constraints of being $\epsilon$-soft, because adjusting any low probabilities to be high enough would mean reducing other probabilities, and there are multiple ways you could do that.

However, a really simple way to make any starting policy $\pi$ into an $\epsilon$-soft variant is to make the policy choice in 2 steps - first step choose between the original policy with probability $(1-\epsilon)$, and with probability $\epsilon$ choose a fixed policy with equal probability for each action. Second step, evaluate whichever policy the first step chose to determine the action.