# Could we update the policy network with previous trajectories using supervised learning?

I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is derived to be an expectation over these trajectories.

Doesn't the following intuition from the OpenAI Vanilla Policy Gradient description indeed propose that learning from prior experience should still be possible?

The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return.

The goal is to change the probabilities of actions. Actions sampled from previous policies are still possible under the current one.

I see that we cannot reuse the previous actions to estimate the policy gradient. But couldn't we update the policy network with previous trajectories using supervised learning? The labels for the actions would be between 0 and 1 based on how good an action was. In the simplest case, just 1 for good actions and 0 for bad ones. The loss could be a simple sum of squared differences with a regularization term.

Why is that not used/possible? What am I missing?

To make things clearer, consider a simple case. Let's say you take action $$a_1$$ and you end up in state $$s_1$$ with reward $$0$$. Then you have two possibilities, you take action $$a_2$$ and end up in terminal state $$s_2$$ with reward $$-10$$ or you take action $$a_2'$$ and end up in terminal state $$s_2'$$ with reward $$10$$. Let's say you use trajectory $$a_1 \rightarrow s_1 \rightarrow a_2 \rightarrow s_2$$ with return $$-10$$ to learn about action $$a_1$$. Then your label for that action would probably be that that action is bad, but it actually isn't, if you took action $$a_2'$$ after $$a_1$$ your return for action $$a_1$$ would be $$10$$. So you learned that your action is bad even though it might not be. Now, if later you learn that taking action $$a_2'$$ is good to take after $$a_1$$ then you would also learn that $$a_1$$ might be good but if you keep using that old data with return $$-10$$ you will keep learning that $$a_1$$ is bad.