In policy gradient methods such as A3C/PPO, the output from the network is probabilities for each of the actions. At training time, the action to take is sampled from the probability distribution.
When evaluating the policy in an environment, what would be the effect of always picking the action that has the highest probability instead of sampling from the probability distribution?
When evaluating the policy in an environment, what would be the effect of always picking the action that has the highest probability instead of sampling from the probability distribution?
Depends what you mean by "evaluating the policy". Unlike in value-based methods, such as Q learning, the policy in gradient methods is not implied by anything else, it is described directly by the probability density function that is being optimised.
Taking the maximum probability item will technically change the policy (unless you are already using deterministic policy gradient), and you would be evaluating a different but related policy to that found by your policy gradient.
However, in a standard MDP environment, and after at least some training, this should be a reasonable process that would give some indication of how well the agent is performing.
In some cases, the nature of the environment means the agent is relying on a stochastic policy. In some partially-observable scenarios it may be better to decide randomly - a simple example is a corridor that needs to be traversed, but where the state features don't give enough information to determine the true direction. A deterministic policy will not be able to traverse the corridor in both directions, but a stochastic policy will get through it both ways, eventually. Another example is in adversarial situations where another agent can learn your agent's policy (the classic version of that being Scissor/Paper/Stone where two ideal opposed agents would learn probability $\frac{1}{3}$ for each action according to Nash equilibrium)
If you don't think you have these special cases, then it should be OK to derive a deterministic policy from your policy gradient agent, and assess that. That's not quite the same as assessing the "learned policy", but is quite sensible to do once you think the agent has converged anyway, since it may still be selecting non-optimal actions at some low probability, and you could get closer to optimal behaviour by removing that.
