In my related question, I asked about the one step actor critic from The Reinforcement Learning Book by Richard Sutton et al, section 13.5:

The learning is becoming less significant as the episode progresses, by means of making $I$ smaller by the discounting factor, as the episode progresses.

How would this discounting generalize to experience replay?

Meaning, if we want to update $\theta$ and $w$ by some experience e=$(s, a, r, s')$, for which we don't know by how much to discount, how would we accomplish a correct update?

Should we remember the discount amount $I$ in the experience?

Please note the critic here is different from the critic here, because it estimates the state-value function $V(s)$, rather than the action-state-value function $Q(s,a)$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.