I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-1,1] to [0, 1.57] before an action is performed. My question is, when we store the transition $(s_t, a_t, r_t, s_{t+1})$ in replay buffer $R$, should $a_t$ be in [-1,1] or [0, 0.157]
4
-
$\begingroup$ I think it would make more sense to have your network use a sigmoid activation and then multiply the action by 1.57 as part of the network, and then everything should work out. $\endgroup$– DavidCommented May 25, 2022 at 13:27
-
$\begingroup$ @DavidIreland tanh is used in order to have an action distribution (policy) possibly centered around zero. $\endgroup$– Rémy Hosseinkhan BoucherCommented Jul 26, 2022 at 20:45
-
1$\begingroup$ @RémyHosseinkhanBoucher yes, I am fully aware. Here, OP doesn’t have a symmetric action space so it would make more sense to use sigmoid as this outputs in $(0, 1)$ and you can multiply the output by $x$ to get an action in the range $(0, x)$ $\endgroup$– DavidCommented Jul 26, 2022 at 21:00
-
$\begingroup$ @DavidIreland ok sorry:) $\endgroup$– Rémy Hosseinkhan BoucherCommented Jul 27, 2022 at 13:26
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
It has an obvious answer: Network is conditioned to use tanh activation. Hence the action values in the buffer should be in the range [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines for my learning, I overlooked this detail.
-
$\begingroup$ Can you give some details: "the buffer should be in [-1, 1] what do u mean $\endgroup$ Commented Jul 26, 2022 at 20:54
-
$\begingroup$ @rémy-hosseinkhan-boucher that is the action value range in buffer $\endgroup$ Commented Jul 27, 2022 at 8:14