Why does $\alpha=1$ mean batch MC Learning?

Question

Here is a part of slide 4 from the link:

https://tao.lri.fr/tiki-download_wiki_attachment.php?attId=1683

Why does $\alpha=1$ mean batch MC Learning? I do not see this clearly when I compare with averaging returns formula.

Answer 1

I did not read the link but I am giving a standard derivation as it can be found in Sutton-Barto for instance.

The average return formula can be reformulated: \begin{align} V_{n+1}(s) &= \frac{1}{n}\sum_{i=1}^{n}R_{i} \\ &= \frac{1}{n}\left(R_{n} +\sum_{i=1}^{n-1}R_{i} \right) \\ &= \frac{1}{n}\left(R_{n} +(n-1) \frac{1}{n-1}\sum_{i=1}^{n-1}R_{i} \right) \\ &= \frac{1}{n}\left(R_{n} + (n-1) V_{n}(s) \right) \\ &= \frac{1}{n}\left(R_{n} + nV_{n}(s) - V_{n}(s) \right) \\ &= V_{n}(s) + \frac{1}{n} (R_{n} - V_{n}(s)) \end{align} where $V_{n}$ is the estimate of the value function after $n-1$ averages over the return of the state $s$. We can interpret each $i$ as a single visit to $s$ or as a batch of visits of $s$ in which case $R_{i}$ would be the averaged return of this batch. Setting $\frac{1}{n} = \alpha$ directly results in the formula for incremental updates. Note that if we set $n = \alpha = 1$ we just set the value function of the state to the return of the first episode or the average of the first batch $V(s) = R_{1}$.