I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy.

Lillicrap et al. denote the policy gradient by

$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu) \right],$$

where $\beta$ denotes the behavior policy (equation 5 in the original paper).

However, when they talk about exploration, they denote the exploration policy by $ \mu'$. This notation seems confusing to me since the target actor network is also denoted by $\mu'(s|\theta^{\mu'})$.

As far as I understand, the exploration policy is not directly linked to the target critic network but rather corresponds to the previously mentioned behavior policy $\beta$. Is this correct or am I understanding it wrong?

  • $\begingroup$ I think you mean target actor network, not the target critic $\endgroup$ – David Ireland Mar 16 at 17:20
  • $\begingroup$ Did I answer your question? If not then please let me know and I will amend my answer based on your feedback, if so then please accept the question. $\endgroup$ – David Ireland Mar 18 at 10:03
  • $\begingroup$ Thank you a lot for your explanations. Just to make sure I understand it correctly: The tuple $(s,a,r,s')$ was sampled with the (back then valid) exploration policy. However, the current exploration policy might have changed. Therefore, the authors have to introduce a new variable $\beta$ denoting "the policies, that were valid at the time back then". $\endgroup$ – Manuel Mar 18 at 12:25
  • $\begingroup$ @DavidIreland Yes, I was referring to the actor network. I edited the original post. Thank you $\endgroup$ – Manuel Mar 18 at 12:27
  • $\begingroup$ $\beta$ is not a variable, it is a policy. The introduction of $\beta$ means that the tuple $(s, a, r, s')$ does not need to be sampled according to our most recently policy, so we can sample previous experiences that come from the old policy as you say. $\endgroup$ – David Ireland Mar 18 at 14:41

You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either.

Essentially when they take the expectation with respect to $\rho^\beta$ they are taking expectation with respect to a state distribution induced by some policy $\beta$ that is not necessarily the same as our current policy -- this is what makes DDPG an off-policy algorithm.

The target actor network $\mu'$ is used in the loss function for the critic; the target for the critic (ignoring parameters for brevity) $y = r + \gamma Q(s', \mu'(s'))$ where $s'$ is the state we transitioned to from $s$ when we took an action $a$.

Now, $a$ was sampled as part of the tuple $(s, a, r, s')$ from our replay buffer, meaning that $a$ will have been chosen according to some past version of our policy, plus some exploration noise $\mathcal{N}$. The way this links to the behaviour policy $\beta$ is that because we have sampled it from some old version of our policy, i.e. not our current policy, it is instead coming from this behaviour policy $\beta$.

The target network $\mu'$ is simply a copy of the current actor network where the weights are updated using the polyak averaging technique and is not really related to the behaviour policy $\beta$, at least not in any useful way for you to think about it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.