Slide 11, https://cs.uwaterloo.ca/~ppoupart/teaching/cs885-spring18/slides/cs885-lecture15b.pdf
Why is $t$ included under the expectation? Normally, instead of "t", I would expect $(a_t,s_t) \sim \pi_{\theta_{old}} $.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In practice policy optimization algorithms like TRPO collect trajectories of experiences by interacting with the environment under the current policy, thus the sloppy notation here with expectation over $t$ just signifies that the empirical estimate of the surrogate objective is taken over all time steps in the collected trajectories. Including just $t$ under the expectation is a simpler and more compact though sloppy and unconventional way to convey this, while your proposal is usually computationally efficient to sample local state-action pairs from a batch of collected experiences replay buffer rather than needing to consider entire trajectories such as DQN, PPO, A2C, etc.
