In Q-learning with function approximators such as Neural Networks, we typically implement a memory so that at the end of each episode we also train on past experiences. This is typically fine because Q-learning is off-policy learning. Is it the case that this would not work with the REINFORCE algorithm? Is there a "mathematical" reason why this would be the case? I can't figure if "off-policy" learning would be detrimental to REINFORCE?
$\begingroup$
$\endgroup$
asked May 17 at 10:59
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
16
Using a "memory" of previous experience with the REINFORCE algorithm will not work. The algorithm relies heavily on the training data distribution matching current on-policy behaviour.
Learning the optimal policy in REINFORCE can be framed as a race where whatever the current policy choice is, is treated as correct, weighted only by return value and distribution. Using a history for training will either select that whole history as correct, or reject it (constructing some kind of probably non optimal anti-policy) depending on whether the returns at that time were positive or negative. Those returns will most likely be incorrect as they will be due to combined choices made when the policy was more random. And even if they are correct, if there is selection bias in the distribution, it will very heavily skewed results.
For example, if in state $s$, the history contains two examples of selecting action $a_1$ with return 2 and one of $a_2$ with return 3, then REINFORCE repeatedly trained on this will prefer action $a_1$. This effect can still happen on-policy, but is mitigated by the change in policy that occurs dynamically during on-policy sampling.
There are ways to use policy gradient in off-policy. One is the method Deep Deterministic Policy Gradient (DDPG), where the policy network outputs a single action choice, and a separate noise function is added to that choice to collect experience.
Also actor-critic methods can use off-policy approaches to train the critic, and could use a replay memory for that
-
$\begingroup$ I will need to check whether DDPG is used with a history $\endgroup$ Commented May 17 at 12:00
-
2$\begingroup$ Yes, DDPG is off-policy and uses a replay buffer. @NeilSlater $\endgroup$– DavidCommented May 17 at 13:13
-
$\begingroup$ @NeilSlater For those that might be implementing REINFORCE with baseline though, the second Neural Network that learns the state-value function can implement a memory though, right? I understand the policy network can't thanks to your example but I think it would be ok to have a memory for the state-value NN? Although I'm not sure since if a bad decision is over-represented in the memory it might be problematic? But that argument could be made for Q-learning and we do use a memory in Q-learning. $\endgroup$ Commented May 26 at 9:27
-
$\begingroup$ @FluidMechanicsPotentialFlows Yes that would be similar to the actor-critic approach as long as you use an off-policy learning algorithm for the state value function (assuming you are using state value for the baseline). $\endgroup$ Commented May 26 at 21:17
-
$\begingroup$ What would an off-policy learning algorithm for the state value function be? I was thinking to use a Neural Network that tries to minimise its prediction of the reward at a given state with the reward actually received at this given state, following the REINFORCE policy. Does this have a name? It seems similar to Q-learning but it's not really learning Q-values. $\endgroup$ Commented May 27 at 20:01