Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents

Question

Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for discrete ones it can be a Categorical (or a relaxation).

The question is: during training we sample from such distribution and then compute the log-probability, but during inference (also agent evaluation/test) what is the best to do? Usually, during evaluation we want to exploit so we should't sample the action (unless is recommended for a given environment setup, to avoid an opponent to exploit a deterministic policy - but I'd like to exclude this case), and so we want to take the action that exploits the more.

I'm puzzled about whether to take the mean or mode of the action distribution during inference and evaluation (I assume the agent training has ended): also, is the most likely action the action that exploits?

In pseudo-code you can view the problem as follows:

function act(state):
   action_distribution = policy(state)
   
   if evaluation:
      # here is the problem: mean or mode?
      actions = action_distribution.mean()  # or mode()
   else:
      # training phase
      actions = action_distribution.sample()

  return actions

Clarifications: I assume a generic action distribution in which its mean can be different from its mode, like a Beta distribution for example. What I also ask is:

• Considering a given state, does taking the mean of the distribution corresponds to an action that is associated to the highest expected value?
• Similarly, considering the mode should correspond to the most likely action (to simplify I assume the mode to be unique) which should be associated to the highest state-action (or Q) value. Is that so?

Additionally, I'm aware of this answer: it's related but does not answer my question.

What I'd like to know is: at evaluation/inference time, assuming for a given environment is fine to have a deterministic policy, how to best implement/obtain that from a learned action distribution?

Answer 1

(This does not fully answer your question. Actually, it just attempts to make sure you understood certain things).

The distribution that you learn should put more weight/mass/density on the best actions. So, if you sample during inference, you should get the best actions more frequently than other actions.

You would use the same action every time if you are certain that the optional policy is deterministic, which is the case of finite MDPs.

There are cases where the best policy is really stochastic (e.g. rock-paper-scissors, which is naturally represented as Markov Game, so not a finite MDP), so choosing always the same action is likely not a good idea, so the best approach during inference would actually be to sample.

So, you would need to pick the same action every time if you have a finite MDP (finite action and state spaces).

See also this answer