I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep.

Say, the action is $a = (a_1, a_2, a_3)$. At the start of an episode $i$, the agent uses actions $a_1, a_2$ only at timestep 1. Action $a_3$ is used at every timestep in the episode starting from 1 till the horizon H. The agent receives rewards $r_i$ at each timestep $i$ till the end of the episode.

I am using SAC. Since the actions $a_1, a_2$ only affect the agent's behavior at timestep 1 and are not used at any of the later timesteps, I am not sure if the RL policy will get better at choosing "good" values for $a_1, a_2$.

Will the RL be able to learn a good policy even if it doesn't quite see the effects of the first two actions in the episode data except at the first timestep?


1 Answer 1


You should be able to learn a good policy even if you use the first two actions only at the first timestep.

Using this OpenAI reference, the loss for the state action value function (from which the policy loss is later derived) is:

$$L(\phi) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi_{target})\right)^2\right]$$

where $D$ is a set of transitions, $\phi_{target}$ are "old" parameters for the action state value function which are left unchanged in the parameter update, and $a' \sim \pi(.|s,\theta)$.

Note that I've simplified the equation for clarity.

The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.

At timestep 0, the target $r + \gamma Q(s', a'|\phi_{target})$ for $Q(s_{t_0},a_{t_0}|\phi)$ (with $a' \sim \pi(.|a_{t_0}, \theta)$) in the loss will be non-zero, because $Q(s', a'|\phi_{target})$ will be non-zero and will reflect the value of $(s',a')$ accurately (e.g. thanks to transitions which happen at later timesteps).

  • $\begingroup$ Thank you for your answer. But, I believe I did not make it clear in my question (I have edited it now). I am getting rewards at each timestep but, only use the third action at every timestep and not the first two, which are used only on the first timestep of every episode. My concern is that since the first two actions do not affect the agent's behavior at each timestep, the RL policy might not be good at choosing good values for $a_1$ and $a_2$. $\endgroup$ Aug 4, 2022 at 12:56
  • $\begingroup$ I had understood that the first two actions are only used in the first timestep. When I say "even if you don't see a reward", that wasn't to say that I thought your issue was with having a reward or not, but that even if you didn't see a reward in the first timestep, the policy would be accurate for the action $(a_1, a_2)$ because the target will be non-zero in either case (reward or not). I've amended the answer to be clearer $\endgroup$ Aug 8, 2022 at 18:43
  • $\begingroup$ Thanks! Now I understand. $\endgroup$ Aug 9, 2022 at 19:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .