What is the motivation behind using a deterministic policy?

Question

What is the motivation behind using a deterministic policy? Given that the environment is uncertain, it seems stochastic policy makes more sense.

Answer 1

You're right! Behaving according to a deterministic policy while still learning would be a terrible idea in most cases (with the exception of environments that "do the exploring for you"; see comments). But deterministic policies are learned off-policy. That is, the experience used to learn the deterministic policy is gathered by behaving according to a stochastic behavior policy.

Under some reasonable assumptions--like that the environment is fully-observed and is stationary--an optimal deterministic policy always exists. The proof can be found in chapter 6 of "Markov Decision Process -- Discrete Stochastic Dynamic Programming" by Martin L. Puterman. The same cannot be said for stochastic polices. For this kind of environment (even if it's stochastic) an optimal policy is hardly ever stochastic.

So, a motivation for wanting to learn a deterministic policy is often because we know that there is an optimal deterministic policy.

It's possible your question was also tangentially about off-policy learning. "Why learn a deterministic policy directly (off-policy) when we can just use something like decaying $\epsilon$-greedy?" Briefly, off-policy learning is very powerful and general. It's necessary in any algorithm that uses experience replay, for example. A discussion about the merits of off-policy learning is probably best left to another question, but reading Section 5.5 of Sutton and Barto's RL book should get you started.

Finally, directly learning a deterministic policy could be more computationally efficient if using deterministic policy gradients. In the setting with continuous state and action spaces, the deterministic policy gradient exists and has a simpler expectation than the stochastic policy gradient.

Stochastic policy gradient:

$$\begin{align*} \nabla_\theta J(\pi_\theta) &= \int_\mathcal{S} \rho^\pi (s) \int_\mathcal{A} \nabla_\theta \pi_\theta (a|s) Q^\pi (s,a) \text{d}a\text{d}s\\ & = \mathbb{E}_{s\sim \rho^\pi, a\sim\pi_\theta}[\nabla_\theta \log \pi_\theta (a|s) Q^\pi (s,a)] \end{align*}$$

Deterministic policy gradient: $$\begin{align*} \nabla_\theta J(\mu_\theta) &= \int_\mathcal{S} \rho^\mu (s) \nabla_\theta \mu_\theta (s) \nabla_a Q^\mu (s,a)|_{a=\mu_\theta (s)}\text{d}s\\ & = \mathbb{E}_{s\sim \rho^\mu}[\nabla_\theta \mu_\theta (s) \nabla_a Q^\mu (s,a)|_{a=\mu_\theta (s)}] \end{align*}$$

Notice that the expectation in the deterministic policy gradient isn't over the action space. Estimating this expectation would require many fewer samples in the setting of a continuous, high-dimensional action space.

To recap:

• Optimal policies are often deterministic, not stochastic
• Learning deterministic policies directly (off-policy) is powerful and general
• It can also be more efficient if in a continuous, high-dimensional action space