What is a weighted average in a non-stationary k-armed bandit problem?

Question

In the book Reinforcement Learning: An Introduction (page 25), by Richard S. Sutton and Andrew G. Barto, there is a discussion of the k-armed bandit problem, where the expected reward from the bandits changes slightly over time (that is, the problem is non-stationary). Instead of updating the Q values by taking an average of all rewards, the book suggests using a constant step-size parameter, so as to give greater weight to more recent rewards. Thus:

$$ Q_{n+1} = Q_n + \alpha (R_n - Q_n),$$

where $\alpha$ is a constant between 0 and 1.

The book then states that this a weighted average because the sum of the weights is equal to 1. What does this mean? Why is this true?

Answer 1

The weighted average stands for a linear combination of all values, such that the sum of all weights is 1.

More specifically, if you denote the rewards by a vector $X$, the weighted average will be taking the dot product between $X$ and a vector $W$ such that $0 \le W_i \le 1$ and the sum of all $W_i$ is 1.

If each $W_i = 1/n$ it will be a weighted average (a.k.a the mean). Using the exponential decay $W_i = \alpha^i/ (\sum W_i)$ is also a weighted average.

Then both strategies to compute the Q value use weighted average of previous reward.