From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages increase the probabilities, negative advantages reduce the probabilities.

So, how do we compute the advantages given the real discounted rewards (aggregated rewards from the episode) and a policy network that only outputs the probabilities of actions?

$$A(s,a) = Q(s,a) - V(s) \; ,$$
where $$Q(s,a)$$ is the action-value function and $$V(s)$$ is the state-value function. In theory you could represent these by two different function approximators, but this would be quite inefficient. However, note that $$Q(s,a) = \sum_{s',r} \mathbb{P}(s',r|s,a)(r + V(s') = \mathbb{E}[r + V(s')|a,s]\;,$$ so we can actually use a single function approximation, for $$V(s)$$, to completely represent the advantage function. To optimise this function approximator you would use the returns at each step of the episode as in e.g. the REINFORCE algorithm like you mentioned.