# Is $s_0$ the current state in policy gradients?

As far as I understand from here (source: OpenAI), the objective function in Policy Gradient is:

$$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$

where $$R(\tau)=r_0+r_1+...+r_T$$, with $$r_t$$ being taken from a trajectory $$\tau = (s_0,a_0,s_1,a_1,...)$$. $$s_0$$ is defined as the start state, sampled from a distribution $$\rho_0$$, and thus does not depends on the parameter $$\theta$$ for a policy $$\pi$$.

However, in the earlier tutorial here (source: OpenAI), the definition for the value function $$V^{\pi}$$ is $$V^{\pi}(s)=E_{\tau\sim\pi}[R(\tau)|s_0=s]$$. Compared with the definition in Sutton & Barto, i.e. $$V^{\pi}(s)=E_{\pi}[G_t|s_t=s]$$.

Can I correctly understand that OpenAI is now defining $$s_0$$ as the current state? If so, it conflicts with the earlier definition in the policy gradient setting, where $$s_0$$ must be the start state, so that its probability does not depend on the policy.

• the 'correct' objective for policy gradient algorithms is to maximise the value function from the initial state $s_0$. this is why there should also be a second discount factor in the policy gradient, to discount the state distribution. however, in practice, this rarely happens. you can look at a paper 'is the policy gradient a gradient?' for a more in depth discussion on this. Commented Jan 18 at 15:10
• @David There are more than one valid PG objectives. Episodic and continuing problems will have different definitions of $J$ Commented Jan 23 at 8:27
• @NeilSlater sure but this misunderstanding comes up a lot and is mostly due to the lack of second discount factor that is not added in policy gradient algorithms that people will be familiar with. I'm fairly certain there's a question on this site about why there is a second discount factor in the pseudocode of the Sutton and Barto book which is a common source of confusion Commented Jan 23 at 9:50

The On-Policy Value Function, $$V^{\pi}(s)$$, which gives the expected return if you start in state $$s$$ and always act according to policy $$\pi$$
Therefore there seems no incompatibility issue of notation of $$s_0$$ here which is the initial state of a trajectory $$\tau$$, for any current state $$s$$ you can treat it as an initial state of the reward-to-go trajectory $$\tau$$ from that point on until the end of its episode as mentioned in your first reference.