When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a bunch of episodes using randomly sampled actions (or maybe by following an old policy) one time, and train multiple policies using this set of samples to find the most effective hyperparameters? Every time we train a new policy, does it make sense to replay all the episodes generated by the previous policy? I'm mostly interested in actor-critic methods.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
What you're describing is off-policy learning. A classic example is $Q$-learning, where you follow some policy $\pi$ whilst learning about the greedy policy.
If you're interested in actor-critic methods then a popular off-policy method is the Deep Deterministic Policy Gradient.
