Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents

Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for discrete ones it can be a Categorical (or a relaxation).

The question is: during training we sample from such distribution and then compute the log-probability, but during inference (also agent evaluation/test) what is the best to do? Usually, during evaluation we want to exploit so we should't sample the action (unless is recommended for a given environment setup, to avoid an opponent to exploit a deterministic policy - but I'd like to exclude this case), and so we want to take the action that exploits the more.

I'm puzzled about whether to take the mean or mode of the action distribution during inference and evaluation (I assume the agent training has ended): also, is the most likely action the action that exploits?

In pseudo-code you can view the problem as follows:

function act(state):
action_distribution = policy(state)

if evaluation:
# here is the problem: mean or mode?
actions = action_distribution.mean()  # or mode()
else:
# training phase
actions = action_distribution.sample()

return actions


Clarifications: I assume a generic action distribution in which its mean can be different from its mode, like a Beta distribution for example. What I also ask is:

• Considering a given state, does taking the mean of the distribution corresponds to an action that is associated to the highest expected value?
• Similarly, considering the mode should correspond to the most likely action (to simplify I assume the mode to be unique) which should be associated to the highest state-action (or Q) value. Is that so?

What I'd like to know is: at evaluation/inference time, assuming for a given environment is fine to have a deterministic policy, how to best implement/obtain that from a learned action distribution?

• How could you choose the mean action from a Bernoulli distn? The mean is $p$, whereas the mode is 0 or 1. for a discrete distn, you would usually want to choose the modal action. In a continuous distribution, the mode might be complex to calculate so I’d recommend using the mean. Apr 23 at 22:30
• Sure, the Bernoulli is a bad example (I'll edit that) so assume you have a Beta instead (indeed, the action space is continuous), what you would do in that case? still choosing the modal action? If so, can you motivate further Apr 24 at 8:37
• I think my last comment is still valid. Finding the mode could be non-trivial (though I suppose in a Beta it is will known). So, in this case, the advice would simply to be to use whichever gives the best performance. Apr 24 at 17:12

(This does not fully answer your question. Actually, it just attempts to make sure you understood certain things).

The distribution that you learn should put more weight/mass/density on the best actions. So, if you sample during inference, you should get the best actions more frequently than other actions.

You would use the same action every time if you are certain that the optional policy is deterministic, which is the case of finite MDPs.

There are cases where the best policy is really stochastic (e.g. rock-paper-scissors, which is naturally represented as Markov Game, so not a finite MDP), so choosing always the same action is likely not a good idea, so the best approach during inference would actually be to sample.

So, you would need to pick the same action every time if you have a finite MDP (finite action and state spaces).