1
$\begingroup$

According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:

$$ \pi(a|s) = \begin{cases} \frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } a = \arg\max_{a'} Q(s, a') \\\\ \frac{\epsilon}{|A|} & \text{otherwise} \end{cases} $$

I followed an exercise from

https://github.com/dennybritz/reinforcement-learning/blob/master/TD/Q-Learning%20Solution.ipynb

where the policy is implemented in Python like this:

def policy_fn(observation):
    A = np.ones(nA, dtype=float) * epsilon / nA
    best_action = np.argmax(Q[observation])
    A[best_action] += (1.0 - epsilon)
    return A

I wonder why this implementation is different from:

def policy_fn(observation):
    p = np.random.rand()
    if p > epsilon:
        return np.argmax(Q[observation])
    return np.random.randint(0, nA)

At least, I get different results for the final action-state pairs. In my opinion, the best action gets sampled with a probability of $1 -\epsilon$. Otherwise, each action gets chosen with the same probability.

$\endgroup$
1
  • 2
    $\begingroup$ they look equivalent to me. let's look at what happens: with probability $\epsilon$, an action is chosen uniformly at random. That means that all actions have a probability of $\epsilon / |\mathcal{A}|$ of being selected. However, with probability $1 - \epsilon$, we select the greedy action. So, the probability of the greedy action being chosen is $\epsilon / |\mathcal{A}| + 1 - \epsilon$. Both the functions look to implement the same thing, where the first returns the probability of each action being selected (and thus should then be sampled from) and the latter returns the action directly. $\endgroup$
    – David
    Nov 30, 2023 at 15:06

1 Answer 1

1
$\begingroup$

The two implementations you posted are different, but they do represent the same $\epsilon$-greedy policy.

The first function returns an array A which contains the probabilities of each action choice. In order to get an action choice you need to run something like np.random.choice(np.arange(len(A)), p=A) to sample from it.

The second function directly returns an action choice.

You might want the policy probabilities from the first approach for some kinds of reinforcement learning methods. For instance, importance sampling for off-policy learning needs to divide target policy probability by behaviour policy probability. Having a policy method that returns the full distribution of action choices is a generic way to represent a policy, allowing you to work with the policy in other contexts than taking an action.

You might otherwise prefer the second direct action choice because then you do not need to add extra code to make that choice in the inner loop of the learning algorithm. Technically this is not just representing the policy, but actually sampling from the policy as well.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .