In the book Reinforcement Learning: An Introduction (page 25), by Richard S. Sutton and Andrew G. Barto, there is a discussion of the k-armed bandit problem, where the expected reward from the bandits changes slightly over time (that is, the problem is non-stationary). Instead of updating the Q values by taking an average of all rewards, the book suggests using a constant step-size parameter, so as to give greater weight to more recent rewards. Thus:

$$ Q_{n+1} = Q_n + \alpha (R_n - Q_n),$$

where $\alpha$ is a constant between 0 and 1.

The book then states that this a weighted average because the sum of the weights is equal to 1. What does this mean? Why is this true?


The weighted average stands for a linear combination of all values, such that the sum of all weights is 1.

More specifically, if you denote the rewards by a vector $X$, the weighted average will be taking the dot product between $X$ and a vector $W$ such that $0 \le W_i \le 1$ and the sum of all $W_i$ is 1.

If each $W_i = 1/n$ it will be a weighted average (a.k.a the mean). Using the exponential decay $W_i = \alpha^i/ (\sum W_i)$ is also a weighted average.

Then both strategies to compute the Q value use weighted average of previous reward.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.