I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is derived to be an expectation over these trajectories.
Doesn't the following intuition from the OpenAI Vanilla Policy Gradient description indeed propose that learning from prior experience should still be possible?
The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return.
The goal is to change the probabilities of actions. Actions sampled from previous policies are still possible under the current one.
I see that we cannot reuse the previous actions to estimate the policy gradient. But couldn't we update the policy network with previous trajectories using supervised learning? The labels for the actions would be between 0 and 1 based on how good an action was. In the simplest case, just 1 for good actions and 0 for bad ones. The loss could be a simple sum of squared differences with a regularization term.
Why is that not used/possible? What am I missing?