Can RL still learn if part of my actions are only used once, at the beginning of the episode?

Question

I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep.

Say, the action is $a = (a_1, a_2, a_3)$. At the start of an episode $i$, the agent uses actions $a_1, a_2$ only at timestep 1. Action $a_3$ is used at every timestep in the episode starting from 1 till the horizon H. The agent receives rewards $r_i$ at each timestep $i$ till the end of the episode.

I am using SAC. Since the actions $a_1, a_2$ only affect the agent's behavior at timestep 1 and are not used at any of the later timesteps, I am not sure if the RL policy will get better at choosing "good" values for $a_1, a_2$.

Will the RL be able to learn a good policy even if it doesn't quite see the effects of the first two actions in the episode data except at the first timestep?

Answer 1

You should be able to learn a good policy even if you use the first two actions only at the first timestep.

Using this OpenAI reference, the loss for the state action value function (from which the policy loss is later derived) is:

$$L(\phi) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi_{target})\right)^2\right]$$

where $D$ is a set of transitions, $\phi_{target}$ are "old" parameters for the action state value function which are left unchanged in the parameter update, and $a' \sim \pi(.|s,\theta)$.

Note that I've simplified the equation for clarity.

The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.

At timestep 0, the target $r + \gamma Q(s', a'|\phi_{target})$ for $Q(s_{t_0},a_{t_0}|\phi)$ (with $a' \sim \pi(.|a_{t_0}, \theta)$) in the loss will be non-zero, because $Q(s', a'|\phi_{target})$ will be non-zero and will reflect the value of $(s',a')$ accurately (e.g. thanks to transitions which happen at later timesteps).