# Why are these two implementations of the $\epsilon$-greedy policy different?

According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:

$$\pi(a|s) = \begin{cases} \frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } a = \arg\max_{a'} Q(s, a') \\\\ \frac{\epsilon}{|A|} & \text{otherwise} \end{cases}$$

I followed an exercise from

https://github.com/dennybritz/reinforcement-learning/blob/master/TD/Q-Learning%20Solution.ipynb

where the policy is implemented in Python like this:

def policy_fn(observation):
A = np.ones(nA, dtype=float) * epsilon / nA
best_action = np.argmax(Q[observation])
A[best_action] += (1.0 - epsilon)
return A


I wonder why this implementation is different from:

def policy_fn(observation):
p = np.random.rand()
if p > epsilon:
return np.argmax(Q[observation])
return np.random.randint(0, nA)


At least, I get different results for the final action-state pairs. In my opinion, the best action gets sampled with a probability of $$1 -\epsilon$$. Otherwise, each action gets chosen with the same probability.

• they look equivalent to me. let's look at what happens: with probability $\epsilon$, an action is chosen uniformly at random. That means that all actions have a probability of $\epsilon / |\mathcal{A}|$ of being selected. However, with probability $1 - \epsilon$, we select the greedy action. So, the probability of the greedy action being chosen is $\epsilon / |\mathcal{A}| + 1 - \epsilon$. Both the functions look to implement the same thing, where the first returns the probability of each action being selected (and thus should then be sampled from) and the latter returns the action directly. Nov 30, 2023 at 15:06

The two implementations you posted are different, but they do represent the same $$\epsilon$$-greedy policy.
The first function returns an array A which contains the probabilities of each action choice. In order to get an action choice you need to run something like np.random.choice(np.arange(len(A)), p=A) to sample from it.