While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5).
I am not able to understand how this is mathematically justified. It seems intuitive to me that the "reward to go" when sampled through multiple trajectories should be estimated by the state-value function.
I feel this way since nowhere in the objective function formulation or resulting gradient expression do we tie down the first action after reaching a state. Alternatively, when we sample a bunch of trajectories, these trajectories might include different actions being taken from the state reached in timestep $t$.
So, why isn't the estimation/approximation for the "reward to go" the state value function, in which the expectation is also over all the actions that may be taken from that state as well?