Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for discrete ones it can be a Categorical (or a relaxation).
The question is: during training we sample from such distribution and then compute the log-probability, but during inference (also agent evaluation/test) what is the best to do? Usually, during evaluation we want to exploit so we should't sample the action (unless is recommended for a given environment setup, to avoid an opponent to exploit a deterministic policy - but I'd like to exclude this case), and so we want to take the action that exploits the more.
I'm puzzled about whether to take the
mode of the action distribution during inference and evaluation (I assume the agent training has ended): also, is the most likely action the action that exploits?
In pseudo-code you can view the problem as follows:
function act(state): action_distribution = policy(state) if evaluation: # here is the problem: mean or mode? actions = action_distribution.mean() # or mode() else: # training phase actions = action_distribution.sample() return actions
Clarifications: I assume a generic action distribution in which its mean can be different from its mode, like a Beta distribution for example. What I also ask is:
- Considering a given state, does taking the mean of the distribution corresponds to an action that is associated to the highest expected value?
- Similarly, considering the mode should correspond to the most likely action (to simplify I assume the mode to be unique) which should be associated to the highest state-action (or Q) value. Is that so?
Additionally, I'm aware of this answer: it's related but does not answer my question.
What I'd like to know is: at evaluation/inference time, assuming for a given environment is fine to have a deterministic policy, how to best implement/obtain that from a learned action distribution?