As hinted in your reference the convergence of various TD algos lies at the extended Dvoretzky’s theorem (Theorem 1) which involves maximum norm to be applied to the entire state space with potentially numerous finite states in addition to solve the generic stochastic approximation problem as the estimation error converges to 0 w.p.1. Then you only need to transform the TD(0) iterative process to the form of Theorem 1 and also show it satisfies all 4 constraints there to rigorously prove its convergence w.p.1.
For TD(0) the iterative state value update process of all states $S$ in the finite state space for each step $n$ of an episode is shown in OP as (here for generality $α$ may not be constant):
\begin{align*}
V_{n+1}(S) &= V_n(S) + α_n(S)(R_{n+1} + γV_n(S') - V_n(S)) \\
&=(1 − α_n(S))V_n(S) + α_n(S)(R_{n+1} + γV_n(S'))
\end{align*}
Then define the estimation error as $∆_n(S) = V_n(S) − V_π(S)$, where $V_π(S)$ is the state value under policy $π$, we can easily arrive at:
\begin{align*}
∆_{n+1}(S) &= (1 − α_n(S))∆_n(S) + α_n(S)(R_{n+1} + γV_n(S') - V_π(S))
\end{align*}
Then obviously $\beta_n(S)=α_n(S), F_n(S)=R_{n+1} + γV_n(S') - V_π(S)$, and if $α$ is not constant and satisfies your above constraint $\sum_n \alpha_n(S) = \infty$ and $\sum_n \alpha_n(S)^2 < \infty$ for every state of the finite state space, then condition 1), 2) are satisfied. And since $V_π(S)$ is non-random and $R_{n+1}$ is bounded thus condition 4) is easily satisfied. Finally since $V_n(S')=P_πV_n(S)$ where $P_π$ is the state transition probability and $V_π(S)=R_π+ γP_πV_π(S)$, thus we have the conditional expectation value in condition 3) bounded from above as required. Thus we can prove TD(0) convergence.
