2
$\begingroup$

What is the motivation behind using a deterministic policy? Given that the environment is uncertain, it seems stochastic policy makes more sense.

$\endgroup$
3
$\begingroup$

You're right! Behaving according to a deterministic policy while still learning would be a terrible idea in most cases (with the exception of environments that "do the exploring for you"; see comments). But deterministic policies are learned off-policy. That is, the experience used to learn the deterministic policy is gathered by behaving according to a stochastic behavior policy.

Under some reasonable assumptions--like that the environment is fully-observed and is stationary--an optimal deterministic policy always exists. The proof can be found in chapter 6 of "Markov Decision Process -- Discrete Stochastic Dynamic Programming" by Martin L. Puterman. The same cannot be said for stochastic polices. For this kind of environment (even if it's stochastic) an optimal policy is hardly ever stochastic.

So, a motivation for wanting to learn a deterministic policy is often because we know that there is an optimal deterministic policy.

It's possible your question was also tangentially about off-policy learning. "Why learn a deterministic policy directly (off-policy) when we can just use something like decaying $\epsilon$-greedy?" Briefly, off-policy learning is very powerful and general. It's necessary in any algorithm that uses experience replay, for example. A discussion about the merits of off-policy learning is probably best left to another question, but reading Section 5.5 of Sutton and Barto's RL book should get you started.

Finally, directly learning a deterministic policy could be more computationally efficient if using deterministic policy gradients. In the setting with continuous state and action spaces, the deterministic policy gradient exists and has a simpler expectation than the stochastic policy gradient.

Stochastic policy gradient:

$$\begin{align*} \nabla_\theta J(\pi_\theta) &= \int_\mathcal{S} \rho^\pi (s) \int_\mathcal{A} \nabla_\theta \pi_\theta (a|s) Q^\pi (s,a) \text{d}a\text{d}s\\ & = \mathbb{E}_{s\sim \rho^\pi, a\sim\pi_\theta}[\nabla_\theta \log \pi_\theta (a|s) Q^\pi (s,a)] \end{align*}$$

Deterministic policy gradient: $$\begin{align*} \nabla_\theta J(\mu_\theta) &= \int_\mathcal{S} \rho^\mu (s) \nabla_\theta \mu_\theta (s) \nabla_a Q^\mu (s,a)|_{a=\mu_\theta (s)}\text{d}s\\ & = \mathbb{E}_{s\sim \rho^\mu}[\nabla_\theta \mu_\theta (s) \nabla_a Q^\mu (s,a)|_{a=\mu_\theta (s)}] \end{align*}$$

Notice that the expectation in the deterministic policy gradient isn't over the action space. Estimating this expectation would require many fewer samples in the setting of a continuous, high-dimensional action space.

To recap:

  • Optimal policies are often deterministic, not stochastic
  • Learning deterministic policies directly (off-policy) is powerful and general
  • It can also be more efficient if in a continuous, high-dimensional action space
$\endgroup$
  • $\begingroup$ Maybe add second/third bullet points at the end: 2) A stochastic environment does not mean that there is an optimal stochastic policy 3) It is common for there to be no optimal stochastic policy $\endgroup$ – Neil Slater Apr 4 at 19:17
  • 1
    $\begingroup$ I would also like to comment that actually if the environment has enough randomness, it is possible to learn optimal control, on-policy with a deterministic greedy policy. This is how TD-Gammon worked for instance. They key is that the environment adds the necessary state exploration $\endgroup$ – Neil Slater Apr 4 at 19:18
  • $\begingroup$ @NeilSlater Good points. Edits should reflect those changes. $\endgroup$ – Philip Raeisghasem Apr 4 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.