Why exactly was previously believed that the deterministic policy gradient did not exist?

Question

I'm reading the paper Deterministic Policy Gradient Algorithms, David Silver et al.

First of all, in the introduction, the author says that

It was previously believed that the deterministic policy gradient did not exist

But, I wonder why it is. The general version of the policy gradient theorem does not have restrictions about the policy $\pi$. So, if we choose the policy $\pi$ as a Dirac measure, that is, $\pi(\cdot | s) = \delta_{a}$ for some $a \in \mathcal{A}$, then it is exactly the notion of deterministic policy, so we can apply the usual gradient descent theorem.

Indeed, in theorem 2, they showed that deterministic policy gradient theorem and usual gradient theorem matches when it comes to zero variance. (In fact, I can't understand the statement rigorously, because the policy is something about probability "measure", and the variance is something about "random variable".) However, my below computation shows some contradiction.

Let $\pi(\cdot | s) = \delta_a$, a Dirac measure for some atom $a \in \mathcal{A}$. Following the notation of the paper DPG, a policy gradient theorem says

$$\nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{s, a}[\nabla_{\theta}\log \pi_{\theta}(a | s) * Q^{\pi}(s, a)] $$

A definition of integral shows $$\nabla_\theta J(\pi_{\theta}) = \int_{\mathcal{S}, \mathcal{A}} \pi_{\theta}(da|s) \rho^{\pi}{(ds)} \nabla_\theta \log \pi_\theta(a|s) * Q^{\pi}(s,a). $$ But since $\pi_\theta$ has only an atom at generic element $a \in \mathcal{A}$, so it is same with $$\int_\mathcal{S} \rho^{\pi}(s) \nabla_\theta \log \pi_\theta(a | s) * Q^{\pi}(s, a) = \mathbb{E}_s[\nabla_\theta \log \pi_\theta(a|s) * Q^{\pi}(s, a)].$$

(NOTE : the last term seems the same with the first line of the equation, but we get rid of $a$ from the expectation, by fixing $a$ corresponding to the atom of $s$.) However, note that $\log\pi_\theta(a|s) = 1$, since $\pi_\theta(a|s) = \delta_a(\{a\}) = 1$ as we defined! Thus, $\nabla_\theta \log \pi_\theta(a | s) = \nabla_\theta 1 = 0$ so we reached that gradient vanishes..

However, clearly not.

Can anyone help me?

Answer 1

Actually, your result that the gradient is 0 is correct given your formulation. Indeed, that is why one might have believed that the deterministic policy gradient didn't exist.

The term $\nabla_{\theta}\log \pi(a \, | \, s)$ is a type of gradient estimator known as a likelihood ratio, and it assumes that the support of $\pi$ does not depend on $\theta$. In other words there must be some non-zero probability of choosing all the possible actions $\mathcal{A}$, and $\theta$ encodes those probabilities. In your idea, $\mathcal{A}$ is a single action, and therefore the parameters don't do anything at all. You always have probability 1 of choosing that action. We can't let different parameters choose different actions, as that violates the assumptions of the likelihood ratio and will lead to a biased gradient.

But why can't we let the support of a likelihood ratio depend on $\theta$? I won't prove this in a rigorous way here but you can consider \begin{align*} \mathbb{E}_{y} [ \nabla_{\theta}\, \log p_{\theta}(y)] = \mathbb{E}_{y} \left[ \underset{\epsilon \rightarrow 0}{\lim} \dfrac{\log p_{\theta+\epsilon}(y) - \log p_{\theta}(y)}{\epsilon}\right] \end{align*} and see that if $p_{\theta + \epsilon}(y) = 0$, then $\log p_{\theta+\epsilon}(y) = -\infty$.

I think a more instructive example is to consider what can go wrong if we just go ahead and let the support depend on $\theta$ anyway. Let's consider $\nabla_{\theta} \, \mathbb{E} [u]$ where $u \sim \text{Uniform}(0, \theta)$ (so $\theta$ is a scalar, and $u$ has support on $[0, \theta ]$). We know $\mathbb{E} [u] = \frac{1}{2}\theta$ so $\nabla_{\theta}\mathbb{E} [u] = \frac{1}{2}$. You should work out for yourself that the gradient estimator (using a likelihood ratio) gives the utterly wrong result of $\mathbb{E}[\nabla_\theta \log p(u)] = -1/\theta$