Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below:
Say you have a reward array of length
n and a discount hyperparam
gamma. You might calculate the discount factors like:
gamma^i for i in range(n)
ex: [0.99^0, 0.99^1, ... 0.99^n]
Then get the discounted rewards by multiplying each discount factor by the corresponding reward and doing a cumulative sum.
However, this results in an array of the same length as rewards, but this seems incorrect to me?
I understand discounted rewards as a way to deal with the uncertainty of past actions on future awards. It would seem to me that uncertainty should result in multiple different contexts and then multiple different arrays with their own discounts.
For instance, say I have 4 states with 4 rewards that looks like [2, 3, 1, 3]. It would seem to me I should then have 4 reward arrays:
[2, 3, 1, 3]
[3, 1, 3]
and I should have 4 discount factor arrays also:
[0.99^0, 0.99^1, 0.99^2, 0.99^3]
[0.99^0, 0.99^1, 0.99^2]
because of the different contexts. For instance, in the first state reward value 2 is certain and reward value 3 slightly less so. But once we're in the second state, reward value 3 is certain and reward value 1 slightly less so.
If we don't do something like this, isn't our agent being updated on a system that always keys on the first state?