2
$\begingroup$

In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be expressed as

$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu) \right]\;.$$

We are interested in learning about the policy parameters of $\mu$, denoted by $\theta$, but we take expected with respect to some discounted state distribution induced by a policy $\beta$, which we will denote as $\rho^\beta$.

To summarise, we are learning off-policy as the expectation of the gradient is taken with respect to some state distribution that occurs under some policy that we are not learning about.

Given that on-policy learning is a special case of off-policy learning, if the replay buffer had a size of one, i.e. we use only the most recent experience tuple to perform parameter updates, then DDPG would be on-policy.

Improve this answer
$\endgroup$
0
$\begingroup$

If there was no action noise it would probably not explore enough to obtain a good estimate of Q or the policy gradient.

Instead of estimating Q of the target policy you could estimate Q of the behavior policy but then you have a stochastic policy and the deterministic policy gradient theorem does not work anymore as it is a special case of the stochastic policy gradient theorem (see Section 3.3 of the DPG paper, http://proceedings.mlr.press/v32/silver14.pdf). You would have to use policy gradient theorem of Section 2.2 from the DPG paper.

Improve this answer
$\endgroup$
11
  • $\begingroup$ Thanks for your answer, what is the behaviour policy in DDPG? $\endgroup$
    – GoingMyWay
    Commented Apr 30, 2020 at 4:22
  • $\begingroup$ In algorithm 1 from the DDPG paper (arxiv.org/pdf/1509.02971.pdf) you find the behavior policy in line 8 (Select action a_t ...). It is the target policy plus some exploration noise. For this paper they used an "Ornstein-Uhlenbeck process" (last paragraph of Section 3 and Appendix 7). It seems like Gaussian noise works, too (source: spinningup.openai.com/en/latest/algorithms/…). $\endgroup$
    – alfa
    Commented Apr 30, 2020 at 7:27
  • $\begingroup$ I think that is not a behaviour policy right? Even on-policy algorithms like PPO has such policy. $\endgroup$
    – GoingMyWay
    Commented Apr 30, 2020 at 12:58
  • 1
    $\begingroup$ OK, you could say that without exploration noise it is on-policy (with a deterministic policy). It would most likely not work though. If you had an infinite amount of samples, infinite capacity of the neural networks, and infinite training time for your networks you would perfectly fit Q of mu and I guess mu would get stuck in some local optimum. Theoretically this is not much different from tabular Q-learning. $\endgroup$
    – alfa
    Commented May 5, 2020 at 15:34
  • 1
    $\begingroup$ I think that although this does not answer the posed question directly, it is the correct answer. DDPG without noise is at best a partially functioning RL algorithm, and probably not of much interest. It may work successfully in environments with stochastic overlapping state transitions. TD Gammon for example got away with only greedy action choice in a SARSA/Q-learning algorithm due to randomness inherent in the game $\endgroup$
    – Neil Slater
    Commented Aug 20, 2020 at 12:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .