5
$\begingroup$

I've seen numerous mathematical explanations of reward, V(s) value functions, and returns functions. The reward provides an immediate return for being in a specific state. The better the reward, the better the state.

As I understand it, it can be better to be in a low-reward state sometimes because we can accumulate more long term which is where the expected returns function comes in. An expected return, return or cummulative reward function effectively adds up the rewards from the current state to the goal state. This implies it's model-based. However it seems a Value function does exactly the same?

Is a Value function a Return function? Or are they different?

$\endgroup$
  • $\begingroup$ Why did you revert by edit? I tried to improve the clarity of your question. Now, given that it is hard to understand it, I downvote it (because I don't want to lose 10 minutes again to decipher what you are talking about). $\endgroup$ – nbro Feb 15 at 11:28
  • $\begingroup$ @nbro sorry but I don't care about votes. You changed it to the point where was too generic. The terms I've used are found in common books. I know you want the points but I'd rather the question be helpful to people struggling as I was. Students use this site who aren't as clear as yourself on whether something referred to in a book is EXACTLY the same as another - the adjectives help a great deal. Those who understand will figure out what I'm talking about. Those who don't may get here BECAUSE of the keywords and understand the answer below. $\endgroup$ – user3168961 Feb 15 at 13:04
  • $\begingroup$ I've never seen terms like "expected returns function", "cummulative reward function", or sentences like "we can accumulate more long term" (which is quite nonsensical). I've already read some literature and I don't recall these terms, which I think you either read them from some web article (which you shouldn't be reading in the first place, if that's the case) or you made them up. Can you please tell us exactly where you found these terms?? $\endgroup$ – nbro Feb 15 at 13:06
  • $\begingroup$ Yes, I tried to make your question as general and thus useful as possible. As it is now, it's very specific to your case, so it will not be useful to a lot of people. $\endgroup$ – nbro Feb 15 at 13:09
  • $\begingroup$ @nbro I found refs to "Expected Reward" in Reinforcement Learning by Richard S. Sutton. Otherwise, probably the university assignment I had. I searched at the time and I found the descriptions vague. This answer helped me. Otherwise I don't know. Why would someone make up words? I wasn't lonely dude, I was just struggling with RL :-) $\endgroup$ – user3168961 Feb 15 at 13:27
3
$\begingroup$

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a specific state. A value function is not a "return function", it is an "expected return function" and that is an important difference.

A return is a measured value (or a random variable, when discussed in the abstract) representing the actual (discounted) sum of rewards seen following a specific state or state/action pair.

Typically there is no need to express an individual return as a "return function", although you may find many formulae in RL for sampling or estimating specific return values in order to calculate targets or errors for the value function.

A return (or cumulative reward) function effectively adds up the rewards from the current state to the goal state. This implies it's model-based.

If you have a simple MDP, already accurately modelled, where you can calculate expected return directly from that model, then, yes, in theory, that would be a value function. However, this could be more computationally intensive to resolve than dynamic programming (e.g. Policy Iteration or Value Iteration), and in many cases you don't have any such model, but can still apply RL approaches to learn a value function from experience.

$\endgroup$
  • $\begingroup$ so in DP, return function must use data from existing trials only to calculate value for a state signal? the value function can estimate rewards for actions not yet taken on explored states? $\endgroup$ – user3168961 Mar 18 '18 at 10:45
  • $\begingroup$ @user3168961: Yes, sort of. There are individual returns, which you can experience or sample. In practice, you may use some kind of function for that, which may blur the line between expected and actual return when bootstrapping as in value iteration or TD learning etc. Then there are expected returns, which are learned by the value functions. $\endgroup$ – Neil Slater Mar 18 '18 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.