You should be able to learn a good policy even if you use the first two actions only at the first timestep.

Using [this OpenAI reference](https://spinningup.openai.com/en/latest/algorithms/sac.html), the loss for the state action value function (from which the policy loss is later derived) is:

$$L(\phi) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi_{target})\right)^2\right]$$

where $D$ is a set of transitions, $\phi_{target}$ are "old" parameters for the action state value function which are left unchanged in the parameter update, and $a' \sim \pi(.|s,\theta)$.

Note that I've simplified the equation for clarity.

The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.

At timestep 0, the target $r + \gamma Q(s', a'|\phi_{target})$ for $Q(s_{t_0},a_{t_0}|\phi)$ (with $a' \sim \pi(.|a_{t_0}, 
\theta)$) in the loss will be non-zero, because $Q(s', a'|\phi_{target})$ will be non-zero and will reflect the value of $(s',a')$ accurately (e.g. thanks to transitions which happen at later timesteps).