I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy.

Lillicrap et al. denote the policy gradient by

$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu) \right],$$

where $\beta$ denotes the behavior policy (equation 5 in the original paper).

However, when they talk about exploration, they denote the exploration policy by $ \mu'$. This notation seems confusing to me since the target actor network is also denoted by $\mu'(s|\theta^{\mu'})$.

As far as I understand, the exploration policy is not directly linked to the target critic network but rather corresponds to the previously mentioned behavior policy $\beta$. Is this correct or am I understanding it wrong?

  • $\begingroup$ I think you mean target actor network, not the target critic $\endgroup$ Mar 16, 2021 at 17:20
  • $\begingroup$ Did I answer your question? If not then please let me know and I will amend my answer based on your feedback, if so then please accept the question. $\endgroup$ Mar 18, 2021 at 10:03
  • $\begingroup$ Thank you a lot for your explanations. Just to make sure I understand it correctly: The tuple $(s,a,r,s')$ was sampled with the (back then valid) exploration policy. However, the current exploration policy might have changed. Therefore, the authors have to introduce a new variable $\beta$ denoting "the policies, that were valid at the time back then". $\endgroup$
    – Manuel
    Mar 18, 2021 at 12:25
  • $\begingroup$ @DavidIreland Yes, I was referring to the actor network. I edited the original post. Thank you $\endgroup$
    – Manuel
    Mar 18, 2021 at 12:27
  • $\begingroup$ $\beta$ is not a variable, it is a policy. The introduction of $\beta$ means that the tuple $(s, a, r, s')$ does not need to be sampled according to our most recently policy, so we can sample previous experiences that come from the old policy as you say. $\endgroup$ Mar 18, 2021 at 14:41

1 Answer 1


You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either.

Essentially when they take the expectation with respect to $\rho^\beta$ they are taking expectation with respect to a state distribution induced by some policy $\beta$ that is not necessarily the same as our current policy -- this is what makes DDPG an off-policy algorithm.

The target actor network $\mu'$ is used in the loss function for the critic; the target for the critic (ignoring parameters for brevity) $y = r + \gamma Q(s', \mu'(s'))$ where $s'$ is the state we transitioned to from $s$ when we took an action $a$.

Now, $a$ was sampled as part of the tuple $(s, a, r, s')$ from our replay buffer, meaning that $a$ will have been chosen according to some past version of our policy, plus some exploration noise $\mathcal{N}$. The way this links to the behaviour policy $\beta$ is that because we have sampled it from some old version of our policy, i.e. not our current policy, it is instead coming from this behaviour policy $\beta$.

The target network $\mu'$ is simply a copy of the current actor network where the weights are updated using the polyak averaging technique and is not really related to the behaviour policy $\beta$, at least not in any useful way for you to think about it.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .