0
$\begingroup$

I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-1,1] to [0, 1.57] before an action is performed. My question is, when we store the transition $(s_t, a_t, r_t, s_{t+1})$ in replay buffer $R$, should $a_t$ be in [-1,1] or [0, 0.157]

$\endgroup$
4
  • $\begingroup$ I think it would make more sense to have your network use a sigmoid activation and then multiply the action by 1.57 as part of the network, and then everything should work out. $\endgroup$
    – David
    May 25, 2022 at 13:27
  • $\begingroup$ @DavidIreland tanh is used in order to have an action distribution (policy) possibly centered around zero. $\endgroup$ Jul 26, 2022 at 20:45
  • 1
    $\begingroup$ @RémyHosseinkhanBoucher yes, I am fully aware. Here, OP doesn’t have a symmetric action space so it would make more sense to use sigmoid as this outputs in $(0, 1)$ and you can multiply the output by $x$ to get an action in the range $(0, x)$ $\endgroup$
    – David
    Jul 26, 2022 at 21:00
  • $\begingroup$ @DavidIreland ok sorry:) $\endgroup$ Jul 27, 2022 at 13:26

1 Answer 1

1
$\begingroup$

It has an obvious answer: Network is conditioned to use tanh activation. Hence the action values in the buffer should be in the range [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines for my learning, I overlooked this detail.

$\endgroup$
2
  • $\begingroup$ Can you give some details: "the buffer should be in [-1, 1] what do u mean $\endgroup$ Jul 26, 2022 at 20:54
  • $\begingroup$ @rémy-hosseinkhan-boucher that is the action value range in buffer $\endgroup$
    – goldfinch
    Jul 27, 2022 at 8:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .