Could we update the policy network with previous trajectories using supervised learning?

Question

I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is derived to be an expectation over these trajectories.

Doesn't the following intuition from the OpenAI Vanilla Policy Gradient description indeed propose that learning from prior experience should still be possible?

The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return.

The goal is to change the probabilities of actions. Actions sampled from previous policies are still possible under the current one.

I see that we cannot reuse the previous actions to estimate the policy gradient. But couldn't we update the policy network with previous trajectories using supervised learning? The labels for the actions would be between 0 and 1 based on how good an action was. In the simplest case, just 1 for good actions and 0 for bad ones. The loss could be a simple sum of squared differences with a regularization term.

Why is that not used/possible? What am I missing?

Answer 1

You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning). The only way to possibly know that is to make labels based on the return that you got from that action but the return is based on an old trajectory with the old policy. The return for that specific action depends on actions that happened after that action in the trajectory and return for those actions change with time.

To make things clearer, consider a simple case. Let's say you take action $a_1$ and you end up in state $s_1$ with reward $0$. Then you have two possibilities, you take action $a_2$ and end up in terminal state $s_2$ with reward $-10$ or you take action $a_2'$ and end up in terminal state $s_2'$ with reward $10$. Let's say you use trajectory $a_1 \rightarrow s_1 \rightarrow a_2 \rightarrow s_2$ with return $-10$ to learn about action $a_1$. Then your label for that action would probably be that that action is bad, but it actually isn't, if you took action $a_2'$ after $a_1$ your return for action $a_1$ would be $10$. So you learned that your action is bad even though it might not be. Now, if later you learn that taking action $a_2'$ is good to take after $a_1$ then you would also learn that $a_1$ might be good but if you keep using that old data with return $-10$ you will keep learning that $a_1$ is bad.

You can only use data gathered from the current policy to learn about it because older data might be outdated.