3
$\begingroup$

Sutton-Barto, page 101, Eq (5.2): Assume that $\pi^\prime$ is the $\epsilon$-greedy policy. Then,

\begin{align} q_{\pi}\big(s,\pi'(s)\big)&= \sum_{a}\pi'(a|s)q_{\pi}(s,a) \\ &= \frac{\varepsilon}{|\mathcal{A}(s)|}\sum_{a}q_{\pi}(s,a)+(1-\varepsilon)\max_{a}q_{\pi}(s,a)\\ & \cdots \end{align}

I am confused by the first line in first equation. My confusion points are as follows.

(1) On the left we have $\pi'(s)$ but on the right $\pi'(a|s)$. Since $\pi^\prime$ is stochastic, how can one write $\pi'(s)$?

(2) Why does the first line in equation follow?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

Sutton and Barto are being a bit loose with notation here.

If you had

  • a random variable $\mathbf{X}$, with individual observable values $x$ drawn from set $\mathcal{X}$
  • a discrete probability function for $\mathbf{X}$: $p(x)$ where $\sum_{x \in \mathcal{X}} p(x) = 1$
  • and a function $f(x): \mathcal{X} \rightarrow \mathcal{Y}$ i.e that maps any $x$ to another type

Then you can process $f(\mathbf{X})$, the same function applied to all possible $x$. This returns the distribution for a new random variable $\mathbf{Y}$ - all the output values of $f(x)$ - with the probability of each value depending on $p(x)$

In Sutton & Barto, they are treating $\pi'(s)$ as a random variable for the action choice, like $\mathbf{X}$ (this is where they are being loose, it's not the deterministic policy). Whilst $\pi'(a|s)$ is equivalent to $p(x)$ in the above description.

Improve this answer
$\endgroup$
11
  • $\begingroup$ In your first item, you wrote three different versions of the letter X. How do you pronounce them to differentiate them from each other? $\endgroup$
    – evaristegd
    Commented Mar 24 at 14:59
  • 1
    $\begingroup$ @evaristegd There is an observable value, a set, and a distribution for the same variable. So you would say "the observed value x", or "the set of all possible x", or "the probability distribution for observing specific x" etc. For concise notation, it makes sense to use some variants of the same letter to show that they are related. I used the font variants that are typically used in S&B. $\endgroup$
    – Neil Slater
    Commented Mar 24 at 15:03
  • 1
    $\begingroup$ i would also say that makes no sense to define Q using a distribution as action, because that’s literally the definition of $V^\pi(s)$… $\endgroup$
    – Alberto
    Commented Mar 24 at 18:53
  • $\begingroup$ @Alberto except that is the point of the proof for Policy Improvement Theorem over epsilon-greedy policies, that this is from. It's showing it in a similar way to the deterministic policy version. $\endgroup$
    – Neil Slater
    Commented Mar 24 at 19:37
  • $\begingroup$ @Alberto - also, the value that results is neither $V^{\pi}(s)$ nor $V^{\pi'}(s)$, but something in-between. The proof eventually rolls out the change of policy over the whole trajectory, transforming $V^{\pi}(s)$ to $V^{\pi'}(s)$ and demonstrating $V^{\pi}(s) \leq V^{\pi'}(s)$ $\endgroup$
    – Neil Slater
    Commented Mar 24 at 19:40
2
$\begingroup$

LHS $\pi’(s)$ is just $\pi’(.|s)$ which denotes the whole probability distribution of the said policy given state $s$, while your RHS $\pi’(a|s)$ is just a specific conditional probability when the random variable A=a, given the same state $s$. Once you understand this, you can also immediately see the reason for the first line of your equation.

Improve this answer
$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .