# How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm

After reading Sutton and Barto, I was able to understand the derivation of this theorem. The only thing I don't get is the following part from REINFORCE algorithm:

How are these terms equivalent, and how did $$s$$ get replaced with $$S_t$$?

As far as I understand, the first term is the the marginal expectation of the function

over $$s$$, and hence can be written like the second term.

Also, I'm assuming the proportionality constant gets absorbed in the step size and hence we get rid of the proportionality sign.

• re: the proportionality becoming an equals, you should read like this: the gradient is proportional to the RHS on the top line, and this is in turn equal to the RHS on the second line. So, the gradient is still proportional to the RHS on the second line, the equals is just meaning that it is equal to the top line RHS. To see why it is equal, just not that we are just re-writing the sum as an explicit expectation (which it is, because $\mu(s)$ is a distribution). Commented Jul 19 at 8:46

Your intuition is correct as in your reference just above this equation they explain as:

Notice that the right-hand side of the policy gradient theorem is a sum over states weighted by how often the states occur under the target policy $$\pi$$; if $$\pi$$ is followed, then states will be encountered in these proportions.

And the definition of the distribution $$\mu(s)$$ is also defined at the bottom of the previous page of your quoted section.

• Thanks for the explanation. But what is S_t here and why did we substitute s in that expression? Commented Jul 19 at 12:05
• The added subscript $t$ is to express the parameter update rule of the next equation (13.7) in the same page at any generic time step. There's exact definition linking rv $S_t$ and any generic state $s$ in the beginning of this chapter $\pi(a | s,\mathbf{\theta})=\text{Pr(}A_t =a | S_t =s, \mathbf{\theta}_t =\mathbf{\theta})$. Hope this clarifies your remaining confusion. Commented Jul 19 at 20:05
• It does. Thank you for the explanation. Commented Jul 20 at 16:43

In general, suppose we have a discrete random variable $$X$$. The expectation of $$X$$ is defined as $$\mathbb{E}[X] = \sum_{x\in \mathcal{X}}x \times \mathbb{P}(X = x) \; ;$$ where $$\mathcal{X}$$ is the support of $$X$$.

Given that $$\mu(s)$$ is the state distribution, it is equivalent to say that $$\mathbb{E}\left[\sum_a q_\pi (S_t, a) \nabla \pi(a|S_t, \theta)\right] = \sum_s \mu(s) \left(\sum_a q_\pi (S_t, a) \nabla \pi(a|S_t, \theta)\right) \; ;$$ since $$\mu(s)$$ is the state distribution and $$S_t$$ is used to denote the state as a random variable.