# Are actions deterministic during testing in continuous action space PPO?

In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $$\mu$$ and $$\sigma$$. This results in an inherent exploration. However, during testing, when we no longer need to explore but exploit, the action should be deterministic, i.e. just $$\mu$$, right?

• Which specific RL algorithm are you referring to? Please, edit your post to say this explicitly and provide a link to the paper. Moreover, please, put your specific question in the title. "Action selection during testing in continuous action space" is not a question but a topic/problem. – nbro Jan 9 at 16:53
• Proximal Policy Optimization would be one example. So, some algorithm where the policy network generates action values instead of state-value estimates or the like. – Daniel B. Jan 9 at 16:56
• @DanielB. Ok, let's see what the OP is referring to. I think that a little bit more of context is needed to answer this question properly. – nbro Jan 9 at 17:02
• @nbro, I meant all algorithms that sample their actions using the given $\mu$ and $\sigma$, i.e. Reinforce, A2C, A3C, TRPO, PPO, SAC unlike DDPG and TD3 which return just $\mu$ as their action. But for clarity I edited the my question and made it for PPO. And thank you for improving my question. – Mika Jan 9 at 22:05
• Btw, a thorough explanation of the procedure for sampling actions from a distribution parameterized by a network's predictions/outputs is provided in Section 6 of this paper. Might be that this paper also contains the answer to your question. Haven't read it to the end yet, though. So, I can't tell. – Daniel B. Jan 10 at 2:10