Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for discrete ones it can be a Categorical (or a relaxation).

The question is: during training we sample from such distribution and then compute the log-probability, but during inference (also agent evaluation/test) what is the best to do? Usually, during evaluation we want to exploit so we should't sample the action (unless is recommended for a given environment setup, to avoid an opponent to exploit a deterministic policy - but I'd like to exclude this case), and so we want to take the action that exploits the more.

I'm puzzled about whether to take the mean or mode of the action distribution during inference and evaluation (I assume the agent training has ended): also, is the most likely action the action that exploits?

In pseudo-code you can view the problem as follows:

function act(state):
   action_distribution = policy(state)
   if evaluation:
      # here is the problem: mean or mode?
      actions = action_distribution.mean()  # or mode()
      # training phase
      actions = action_distribution.sample()

  return actions

Clarifications: I assume a generic action distribution in which its mean can be different from its mode, like a Beta distribution for example. What I also ask is:

  • Considering a given state, does taking the mean of the distribution corresponds to an action that is associated to the highest expected value?
  • Similarly, considering the mode should correspond to the most likely action (to simplify I assume the mode to be unique) which should be associated to the highest state-action (or Q) value. Is that so?

Additionally, I'm aware of this answer: it's related but does not answer my question.

What I'd like to know is: at evaluation/inference time, assuming for a given environment is fine to have a deterministic policy, how to best implement/obtain that from a learned action distribution?

  • 1
    $\begingroup$ How could you choose the mean action from a Bernoulli distn? The mean is $p$, whereas the mode is 0 or 1. for a discrete distn, you would usually want to choose the modal action. In a continuous distribution, the mode might be complex to calculate so I’d recommend using the mean. $\endgroup$
    – David
    Commented Apr 23, 2023 at 22:30
  • $\begingroup$ Sure, the Bernoulli is a bad example (I'll edit that) so assume you have a Beta instead (indeed, the action space is continuous), what you would do in that case? still choosing the modal action? If so, can you motivate further $\endgroup$ Commented Apr 24, 2023 at 8:37
  • $\begingroup$ I think my last comment is still valid. Finding the mode could be non-trivial (though I suppose in a Beta it is will known). So, in this case, the advice would simply to be to use whichever gives the best performance. $\endgroup$
    – David
    Commented Apr 24, 2023 at 17:12

1 Answer 1


(This does not fully answer your question. Actually, it just attempts to make sure you understood certain things).

The distribution that you learn should put more weight/mass/density on the best actions. So, if you sample during inference, you should get the best actions more frequently than other actions.

You would use the same action every time if you are certain that the optional policy is deterministic, which is the case of finite MDPs.

There are cases where the best policy is really stochastic (e.g. rock-paper-scissors, which is naturally represented as Markov Game, so not a finite MDP), so choosing always the same action is likely not a good idea, so the best approach during inference would actually be to sample.

So, you would need to pick the same action every time if you have a finite MDP (finite action and state spaces).

See also this answer

  • $\begingroup$ Maybe I should convert this to 1 or more comments, if this doesn't even partially answer your questions. $\endgroup$
    – nbro
    Commented Apr 23, 2023 at 10:26
  • $\begingroup$ Keep as an answer it's interesting. I'm aware of cases like rock-paper-scissor in which the best policy is always stochastic to avoid the opponent to exploit the determinism. But excluding that (I may clarify this in the question), I'd like to understand what is the best to do at inference time with the policy. Actually, could you elaborate a bit more about the difference of finite MDPs and continuous MDPs, in particular why in the continuous case one don't want to have deterministic actions? $\endgroup$ Commented Apr 23, 2023 at 13:06
  • 1
    $\begingroup$ @LucaAnzalone That's actually a good question. I shouldn't have used "only if" but "if" in my last sentence. In other words, I just wanted to point out that, if you have a finite MDP, you're sure that the optimal policy is deterministic. In other cases, it might not be. So, I did not want to say that in continuous actions MDPs the optimal policy is always stochastic, because I don't know if that's the case (I think not - one can probably think of a very simple environment with 2 continuous actions where the optimal policy is deterministic, but I didn't think much about this). $\endgroup$
    – nbro
    Commented Apr 23, 2023 at 17:52
  • $\begingroup$ Well, I can think of an env in which there are three paths that lead to a terminal state, the paths on the left and right are highly rewarding while the middle path is poor. Suppose the actions are steer left/right (from -1 to 1), the "mode" policy will always pick something close to either -1 or 1, while the "mean" policy just average to 0 (or so) leading to the non-rewarding path. In this case the best to do would either to be deterministic or taking one or the modes randomly at each trial. $\endgroup$ Commented Apr 28, 2023 at 17:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .