You should be able to learn a good policy even if you use the first two actions only at the first timestep.

Using [this OpenAI reference](https://spinningup.openai.com/en/latest/algorithms/sac.html), the loss for the state action value function (from which the policy loss is later derived) is:

$$L(\phi) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi_{target})\right)^2\right]$$

where $D$ is a set of transitions, $\phi_{target}$ are "old" parameters for the action state value function which are left unchanged in the parameter update, and $a' \sim \pi(.|s,\phi)$.

Note that I've simplified the equation for clarity.

The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.

Even if you don't see a reward after the first timestep, the target $r + \gamma Q(s', a'|\phi_{target})$ for $Q(s,a|\phi)$ in the loss will be non-zero, because $Q(s', a'|\phi_{target})$ will be non-zero and will reflect the value of $(s',a')$ accurately (e.g. thanks to transitions which happen at horizon $H$ if this is where the environment gives a reward to the agent).