In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent exploration. However, during testing, when we no longer need to explore but exploit, the action should be deterministic, i.e. just $\mu$, right?

  • $\begingroup$ Which specific RL algorithm are you referring to? Please, edit your post to say this explicitly and provide a link to the paper. Moreover, please, put your specific question in the title. "Action selection during testing in continuous action space" is not a question but a topic/problem. $\endgroup$ – nbro Jan 9 at 16:53
  • 1
    $\begingroup$ Proximal Policy Optimization would be one example. So, some algorithm where the policy network generates action values instead of state-value estimates or the like. $\endgroup$ – Daniel B. Jan 9 at 16:56
  • $\begingroup$ @DanielB. Ok, let's see what the OP is referring to. I think that a little bit more of context is needed to answer this question properly. $\endgroup$ – nbro Jan 9 at 17:02
  • 1
    $\begingroup$ @nbro, I meant all algorithms that sample their actions using the given $\mu$ and $\sigma$, i.e. Reinforce, A2C, A3C, TRPO, PPO, SAC unlike DDPG and TD3 which return just $\mu$ as their action. But for clarity I edited the my question and made it for PPO. And thank you for improving my question. $\endgroup$ – Mika Jan 9 at 22:05
  • 1
    $\begingroup$ Btw, a thorough explanation of the procedure for sampling actions from a distribution parameterized by a network's predictions/outputs is provided in Section 6 of this paper. Might be that this paper also contains the answer to your question. Haven't read it to the end yet, though. So, I can't tell. $\endgroup$ – Daniel B. Jan 10 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.