Short answer: I the return-to-go is just a synonym of the reward-to-go, which instead is a well known term used to define the basic form of policy gradient, e.g. REINFORCE-like.
Reward-to-go
As you correctly wrote, the reward-to-go is just a sum of rewards collected in a trajectory $\tau$ from a certain timestep $t$ until termination $T$. If we define $R(\tau)=\sum_{t=1}^T r_t^{\tau}$ as the sum of immediate rewards $r_t$ got in trajectory $\tau$, from that we can define the rewards-to-go just by considering a subset of the trajectory, so discarding the past, i.e starting from a time $t$:
$$R_t(\tau)=\sum_{t'=t}r_{t'}^\tau$$
In particular, the quantity $R_t(\tau)$ figures in the (basic) policy gradient formulation:
$$\nabla_\theta J(\theta)\approx \sum_i\Big(\sum_{t=1}^T \nabla_\theta\log \pi_\theta\big(a_t^{(i)}\mid s_t^{(i)}\big) R_t\big(\tau^{(i)}\big) \Big)$$
Basically, the rewards-to-go score actions within a trajectory $\tau^{(i)}$: the spinning-up rl article you mentioned refers to the fact that the original (although naive) PG formulation uses $R(\tau)$, i.e. sum of rewards, to score the policy's actions but it's useless to consider the immediate rewards $r_k$ gained in the past (so when $k<t$) to score actions happened at time $t$ and onward until the end of the trajectory $T$. In this regard, both $R(\tau)$ and $R_t(\tau)$ are empirical estimates of the value of an action $a_t$ from a given state $s_t$ - think of it as a single-sample estimate of the action-value function, $Q(s_t,a_t)\approx R_t$ - but $R(\tau)$ has a higher variance due the additional sums.
This is reasonable, since in more recent PG formulations we subtract a baseline function $b(s)$ to the rewards-to-go to reduce the variance of the gradient estimate: $\nabla_\theta\log\pi_\theta(a_t\mid s_t)(R_t - b(s_t))$. If $b(s)=V(s)$ is chosen to be the value function, then $R_t - b(s_t)$ is an estimate of the advantage function $A(s_t, a_t) = Q(s_t,a_t) - V(s_t)$ since $R_t$ is the action-value gathered in the current trajectory (but not expected, as in the Q-function.) Formulations that use $A(s,a)$ yield the gradient with lowest variance: A2C uses this.
In which situations do we need to care about it?
We talk about rewards-to-go mainly when dealing with the PG estimation, and defining new baseline functions to improve the gradient estimate: otherwise the term return is largely preferred. Note: it is demonstrated that subtracting a baseline does not alter the gradient formulation, and if correctly picked it does not either add bias to the estimate.
the reward-to-go is analyzed in the paper which states that the expected reward-to-go is the value function
The rewards-to-go are more appropriate as action-value estimates rather than state-value estimates (i.e. the return $G_t$) since we start from a state-action pair. This is a technicality that does not hold in practice, since the value function is also estimated with empirical returns (i.e. the return of a single trajectory $\tau$), i.e. $G_t(\tau) = r_t^\tau + r_{t+1}^\tau +\cdots + r_T^\tau$ that correspond to the rewards-to-go $R_t(\tau)$: think about learning a value function in deep RL, we just fit the neural-net to empirical returns than MSE + GD does the magic. Now, if we average the collected rewards or returns over all possible trajectories we obtain the expected return that is the value function: $V(s_t) = \mathbb{E}_\tau[G_t(\tau)] = \mathbb{E}_\tau[R_t(\tau)] = \mathbb{E}_\tau[r_t^\tau + r_{t+1}^\tau + \cdots + r_{T}^\tau\mid s_t]$.
How is the return-to-go related to the reward-to-go?
They are same thing. I'd say to avoid "return-to-go" and use return, instead (or at most reward-to-go when talking about PG.)
As a side note: none of both terms are mentioned in the famous book "An introduction to reinforcement learning" by Sutton and Barto, so I guess it's something introduced in modern deep RL.
