2
$\begingroup$

As far as I understand from here (source: OpenAI), the objective function in Policy Gradient is:

$$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$

where $R(\tau)=r_0+r_1+...+r_T$, with $r_t$ being taken from a trajectory $\tau = (s_0,a_0,s_1,a_1,...)$. $s_0$ is defined as the start state, sampled from a distribution $\rho_0$, and thus does not depends on the parameter $\theta$ for a policy $\pi$.

However, in the earlier tutorial here (source: OpenAI), the definition for the value function $V^{\pi}$ is $V^{\pi}(s)=E_{\tau\sim\pi}[R(\tau)|s_0=s]$. Compared with the definition in Sutton & Barto, i.e. $V^{\pi}(s)=E_{\pi}[G_t|s_t=s]$.

Can I correctly understand that OpenAI is now defining $s_0$ as the current state? If so, it conflicts with the earlier definition in the policy gradient setting, where $s_0$ must be the start state, so that its probability does not depend on the policy.

Improve this question
$\endgroup$
3
  • $\begingroup$ the 'correct' objective for policy gradient algorithms is to maximise the value function from the initial state $s_0$. this is why there should also be a second discount factor in the policy gradient, to discount the state distribution. however, in practice, this rarely happens. you can look at a paper 'is the policy gradient a gradient?' for a more in depth discussion on this. $\endgroup$
    – David
    Commented Jan 18 at 15:10
  • $\begingroup$ @David There are more than one valid PG objectives. Episodic and continuing problems will have different definitions of $J$ $\endgroup$
    – Neil Slater
    Commented Jan 23 at 8:27
  • $\begingroup$ @NeilSlater sure but this misunderstanding comes up a lot and is mostly due to the lack of second discount factor that is not added in policy gradient algorithms that people will be familiar with. I'm fairly certain there's a question on this site about why there is a second discount factor in the pseudocode of the Sutton and Barto book which is a common source of confusion $\endgroup$
    – David
    Commented Jan 23 at 9:50

1 Answer 1

Reset to default
0
$\begingroup$

Below is the explanation around your stated OpenAI's earlier tutorial for the value function:

The On-Policy Value Function, $V^{\pi}(s)$, which gives the expected return if you start in state $s$ and always act according to policy $\pi$

Therefore there seems no incompatibility issue of notation of $s_0$ here which is the initial state of a trajectory $\tau$, for any current state $s$ you can treat it as an initial state of the reward-to-go trajectory $\tau$ from that point on until the end of its episode as mentioned in your first reference.

Improve this answer
$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .