Timeline for Why is DDPG an off-policy RL algorithm?
Current License: CC BY-SA 4.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 20, 2020 at 12:36 | comment | added | Neil Slater | I think that although this does not answer the posed question directly, it is the correct answer. DDPG without noise is at best a partially functioning RL algorithm, and probably not of much interest. It may work successfully in environments with stochastic overlapping state transitions. TD Gammon for example got away with only greedy action choice in a SARSA/Q-learning algorithm due to randomness inherent in the game | |
| May 5, 2020 at 15:34 | comment | added | alfa | There are no guarantees in Deep RL for anything anyway though, so maybe it works even better than some noisy behavior policies. ;) As far as I understand (I might be wrong here) V-trace guarantees convergence of the value function to "something between target and behavior policy" in the tabular case. With a neural network in the mix I am not sure if anything is guaranteed. V-trace, however, needs a stochastic policy to compute importance sampling weights. mu in DDPG is not stochastic. By the way, in the V-trace paper mu is the behavior policy and pi is the target policy. Another notation. | |
| May 5, 2020 at 15:34 | comment | added | alfa | OK, you could say that without exploration noise it is on-policy (with a deterministic policy). It would most likely not work though. If you had an infinite amount of samples, infinite capacity of the neural networks, and infinite training time for your networks you would perfectly fit Q of mu and I guess mu would get stuck in some local optimum. Theoretically this is not much different from tabular Q-learning. | |
| May 5, 2020 at 1:47 | comment | added | GoingMyWay | Thanks for the reply of the purpose of adding the noise, back to my question: in DDPG, if there are no 𝜖-greedy and no action noise, is DDPG an on-policy algorithm? I am not sure on this point. And I am not clear on if the behaviour is clearly more on the exploration side because I did not find theoretical guarantees on its exploration ability of the behaviour so far. If the behaviour policy is far away from the target policy, it can be a bad policy, some algorithms like Retrace/Vtrace still own some theoretical guarantees on the convergence of the target policy. | |
| May 5, 2020 at 0:05 | comment | added | alfa | The behavior policy is called behavior policy because it defines how the agent behaves. The DDPG agent creates its actions through mu + noise while it approximates Q of mu - its target policy. PPO in its original form uses a stochastic policy and is on-policy. Adding noise would not change that. The stochastic policy ensures sufficient exploration. The purpose of a policy is either eploration or exploration (or both). A behavior policy is clearly more on the exploration side than the target policy. | |
| May 2, 2020 at 1:12 | comment | added | GoingMyWay | I don't think adding noise to actions is a behaviour policy, if that so, in PPO (on-policy), adding a noise to its actions makes it off-policy? It is not the definition. The behaviour policy can be a bad policy, it is used to collect data for target policy to learn. I do not think the main purpose of behaviour policy is used for exploration. | |
| May 1, 2020 at 14:25 | comment | added | alfa | Maybe you have a wrong idea about what off-policy and on-policy means. What would you say is the difference between both? | |
| May 1, 2020 at 14:21 | comment | added | alfa | Yes, it is. In on-policy algorithms the behavior policy is the target policy. The behavior policy is used for exploration and the target policy is used to approximate the value function (Q, V, or A). See Section 5.5 of Sutton & Barto (incompleteideas.net/book/RLbook2020.pdf). | |
| Apr 30, 2020 at 12:58 | comment | added | GoingMyWay | I think that is not a behaviour policy right? Even on-policy algorithms like PPO has such policy. | |
| Apr 30, 2020 at 7:27 | comment | added | alfa | In algorithm 1 from the DDPG paper (arxiv.org/pdf/1509.02971.pdf) you find the behavior policy in line 8 (Select action a_t ...). It is the target policy plus some exploration noise. For this paper they used an "Ornstein-Uhlenbeck process" (last paragraph of Section 3 and Appendix 7). It seems like Gaussian noise works, too (source: spinningup.openai.com/en/latest/algorithms/…). | |
| Apr 30, 2020 at 4:22 | comment | added | GoingMyWay | Thanks for your answer, what is the behaviour policy in DDPG? | |
| Apr 29, 2020 at 17:43 | history | edited | alfa | CC BY-SA 4.0 |
[Edit removed during grace period]
|
| Apr 29, 2020 at 16:41 | review | First posts | |||
| May 3, 2020 at 22:23 | |||||
| Apr 29, 2020 at 16:39 | history | answered | alfa | CC BY-SA 4.0 |