You should be able to learn a good policy even if you use the first two actions only at the first timestep.

Using this OpenAI reference, the loss for the state action value function (from which the policy loss is later derived) is:

$$L(\phi) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi_{target})\right)^2\right]$$

where $D$ is a set of transitions, $\phi_{target}$ are "old" parameters for the action state value function which are left unchanged in the parameter update, and $a' \sim \pi(.|s,\phi)$.

Note that I've simplified the equation for clarity.

The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.

Even if you don't see a reward after the first timestep, the target $r + \gamma Q(s', a'|\phi_{target})$ for $Q(s,a|\phi)$ in the loss will be non-zero, because $Q(s', a'|\phi_{target})$ will be non-zero and will reflect the value of $(s',a')$ accurately (e.g. thanks to transitions which happen at horizon $H$ if this is where the environment gives a reward to the agent).

You should be able to learn a good policy even if you use the first two actions only at the first timestep.

Using this OpenAI reference, the loss for the state action value function (from which the policy loss is later derived) is:

$$L(\phi) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi_{target})\right)^2\right]$$

where $D$ is a set of transitions, $\phi_{target}$ are "old" parameters for the action state value function which are left unchanged in the parameter update, and $a' \sim \pi(.|s,\theta)$.

Note that I've simplified the equation for clarity.

The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.

At timestep 0, the target $r + \gamma Q(s', a'|\phi_{target})$ for $Q(s_{t_0},a_{t_0}|\phi)$ (with $a' \sim \pi(.|a_{t_0}, \theta)$) in the loss will be non-zero, because $Q(s', a'|\phi_{target})$ will be non-zero and will reflect the value of $(s',a')$ accurately (e.g. thanks to transitions which happen at later timesteps).