Questions tagged [reward-to-go]

For questions about the concept of "reward-to-go", which comes up e.g. in the context of policy gradients. The "expected reward-to-go for all states" is sometimes used as a synonym for "value function". See e.g. the paper "Learning the Variance of the Reward-To-Go" (2016) by Aviv Tamar et al. for more details.

Filter by
Sorted by
Tagged with
4
votes
1answer
1k views

Why does the “reward to go” trick in policy gradient methods work?

In the policy gradient method, there's a trick to reduce the variance of policy gradient. We use causality, and remove part of the sum over rewards so that only actions happened after the reward are ...
2
votes
0answers
56 views

What is the return-to-go in reinforcement learning?

In reinforcement learning, the return is defined as some function of the rewards. For example, you can have the discounted return, where you multiply the rewards received at later time steps by ...
2
votes
0answers
125 views

What is the proof that “reward-to-go” reduces variance of policy gradient?

I am following the OpenAI's spinning up tutorial Part 3: Intro to Policy Optimization. It is mentioned there that the reward-to-go reduces the variance of the policy gradient. While I understand the ...
1
vote
1answer
58 views

Why is the “reward to go” replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5). I am not ...