3
$\begingroup$

In policy gradient algorithms the output is a stochastic policy - a probability for each action.

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.

Is it beneficial or is it common to use extra exploration strategies, like UCB, Thompson sampling, etc. with such algorithms?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.

Yes, having a stochastic policy function is the main way that a lot of policy gradient methods achieve exploration, including REINFORCE, A2C, A3C.

Is it beneficial or is it common to use extra exploration strategy like UCB, Thompson sampling etc. n such algorithms?

It can be, but needs to be done carefully, as the gradient sampling for the policy function is different. Many policy gradient methods are strictly on-policy and will not work if you simply add extra exploration. It is relatively straightforward to adjust the critic part of actor-critic methods by using e.g. Q learning update rules for it. However, the gradient of the policy function is trickier.

There are some policy gradient methods that do work with a separate, tunable, exploration function. Deep Deterministic Policy Gradient (DDPG) may be of interest to you - as per the title, it works with a deterministic policy function, and exploration is achieved by adding a separate noise function on top. The sampling for policy gradient is then corrected for being off-policy.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Note this is an area where I have not delved into the maths in any depth, so I cannot help with questions about applicability or implementation of any specific noise function. $\endgroup$
    – Neil Slater
    Jun 7, 2020 at 9:28
  • $\begingroup$ Thank you @Neil for your answer $\endgroup$
    – gnikol
    Jun 7, 2020 at 20:15
2
$\begingroup$

Neil Slater's answer is very nice, but I have a couple more suggestions:

  • You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for more entropy) which should prevent your policy from becoming "too deterministic" too early.

  • You can also try maximum-entropy methods, like SAC, which employ a different strategy for promoting policy entropy.

Improve this answer
$\endgroup$

You must log in to answer this question.