Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?

Question

I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy.

Lillicrap et al. denote the policy gradient by

$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu) \right],$$

where $\beta$ denotes the behavior policy (equation 5 in the original paper).

However, when they talk about exploration, they denote the exploration policy by $ \mu'$. This notation seems confusing to me since the target actor network is also denoted by $\mu'(s|\theta^{\mu'})$.

As far as I understand, the exploration policy is not directly linked to the target critic network but rather corresponds to the previously mentioned behavior policy $\beta$. Is this correct or am I understanding it wrong?

Answer 1

You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either.

Essentially when they take the expectation with respect to $\rho^\beta$ they are taking expectation with respect to a state distribution induced by some policy $\beta$ that is not necessarily the same as our current policy -- this is what makes DDPG an off-policy algorithm.

The target actor network $\mu'$ is used in the loss function for the critic; the target for the critic (ignoring parameters for brevity) $y = r + \gamma Q(s', \mu'(s'))$ where $s'$ is the state we transitioned to from $s$ when we took an action $a$.

Now, $a$ was sampled as part of the tuple $(s, a, r, s')$ from our replay buffer, meaning that $a$ will have been chosen according to some past version of our policy, plus some exploration noise $\mathcal{N}$. The way this links to the behaviour policy $\beta$ is that because we have sampled it from some old version of our policy, i.e. not our current policy, it is instead coming from this behaviour policy $\beta$.

The target network $\mu'$ is simply a copy of the current actor network where the weights are updated using the polyak averaging technique and is not really related to the behaviour policy $\beta$, at least not in any useful way for you to think about it.