0
$\begingroup$

Let $\pi$ be an $\epsilon-soft$ policy with state-action value function $q_{\pi}(s,a)$ and $\pi'$ be an $\epsilon-greedy$ policy with respect to $q_{\pi}$.

In Sutton-Barto RL book (page 101, eq. 5.2), they define

\begin{align} q_{\pi}(s, \pi'(s))=\displaystyle \sum_{a}\pi'(a|s)q_{\pi}(s,a). \end{align} Normally $q_{\pi}(s, a)$ means taking action $a$ at state $s$ and then following the policy $\pi$. Based on this convention, the notation $q_{\pi}(s, \pi'(s))$ is weird because $\pi'(s)$ is not a single action like $a$. I.e., $\pi'$ is a stochastic policy and hence only $\pi'(a|s)$ makes sense.

$\endgroup$
1
  • 2
    $\begingroup$ Rather than writing "Confusion about..." in the title, can you put your specific question in the title? $\endgroup$
    – nbro
    Feb 13, 2022 at 15:27

1 Answer 1

2
$\begingroup$

The book presents a slight abuse of notation where $\pi'(s)$ is shorthand for a distribution of action values described by the more correct $\pi'(a|s)$. At that point there is an implied function composition of $q_{\pi}$ over this distribution, resolved on the right hand side to its expectation.

I believe it does this so that it can briefly show something familiar from the deterministic policy improvement theorem. You could almost read it as "the equivalent of this term (LHS) taken from the previous proof must now be written like this (RHS)".

It would be more correct notation to write something like this:

$$\mathbb{E}[q_{\pi}(s,A')|A' \sim \pi'] = ...$$

or perhaps just start with the right hand side.

$\endgroup$
9
  • $\begingroup$ @ Neil Slater, strangely authors also set $v_{\pi'}(s)=q_{\pi}(s,\pi'(s))$. I cannot see why this holds. $\endgroup$ Feb 13, 2022 at 17:15
  • 1
    $\begingroup$ @DSPinfinity It holds provided the only change between $\pi$ and $\pi'$ is at the single state $s$, and that there are no cyclic paths that allow a revisit to $s$. $\endgroup$ Feb 13, 2022 at 17:27
  • $\begingroup$ @ Neil Slater $v_{\pi'}(s)$ means the value of state $s$ following $\pi'$, $q_{\pi}(s, \pi'(s))$ means value of state state $s$ by applying $\pi'$ and then following $\pi$. So, how can they be equal? Am I missing something? $\endgroup$ Feb 13, 2022 at 17:33
  • 1
    $\begingroup$ They can be equal if the only difference between $\pi$ and $\pi'$ is localised to state $s$ (plus the no loops constraint), because $\pi$ and $\pi'$ are identical for all other states that could follow $s$. $\endgroup$ Feb 13, 2022 at 17:39
  • 1
    $\begingroup$ @DSPinfinity I just re-read section 5.4 that includes that equation and related proof. I cannot see anywhere that it states $v_{\pi'}(s) = q_{\pi}(s, \pi'(s))$ nor anything similar. $\endgroup$ Feb 13, 2022 at 20:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .