# What are the value functions used in reinforcement learning?

In reinforcement learning, we often define two functions, the state-value function

$$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$

and the state-action-value function

$$Q^\pi(s,a) = \mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}\Bigg|S_t=s, A_t=a \right]$$

where $$\mathbb{E}_{\pi}$$ means that these functions are defined as the expectation with respect to a fixed policy $$\pi$$ of what is often called the return, $$\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}$$, where $$\gamma$$ is a discount factor and $$R_{t+k+1}$$ is the reward received from the environment (while the agent interacts with it) from time $$t$$ onwards.

So, both the $$V$$ and $$Q$$ functions are defined as expectations of the return (or the cumulative future discounted reward), but these expectations have different "conditions" (or are conditioned on different variables). The $$V$$ function is the expectation (with respect to a fixed policy $$\pi$$) of the return given that the current state (the state at time $$t$$) is $$s$$. The $$Q$$ function is the expectation (with respect to a fixed policy $$\pi$$) of the return conditioned on the fact that the current state the agent is in is $$s$$ and the action the agent takes at $$s$$ is $$a$$.

Furthermore, the Bellman optimality equation for $$V^*$$ (the optimal value function) can be expressed as the Bellman optimality equation for $$Q^{\pi^*}$$ (the optimal state-action value function associated with the optimal policy $$\pi^*$$) as follows

$$V^*(s) = \max_{a \in \mathcal{A}(s)} Q^{\pi^*}(s, a)$$

This is actually shown (or proved) at page 76 of the book "Reinforcement Learning: An Introduction" (1st edition) by Andrew Barto and Richard S. Sutton.

Are there any other functions, apart from the $$V$$ and $$Q$$ functions defined above, in the RL context? If so, how are they related?

For example, I've heard of the "advantage" or "continuation" functions. How are these functions related to the $$V$$ and $$Q$$ functions? When should one be used as opposed to the other? Note that I'm not just asking about the "advantage" or "continuation" functions, but, if possible, any existing function that is used in RL that is similar (in purpose) to these mentioned functions, and how they are related to each other.

• If you start to include all the ways to estimate return, used in various agents to get better approximations of V and/or Q, then there are dozens of functions, which seems too much for a single answer. Any chance you can limit the scope? E.g. are you only interested in the functions that are theoretical measures of agent performance, and not the many ways they can be derived/approximated? – Neil Slater Feb 14 '19 at 21:08
• @NeilSlater I also thought that this question could eventually lead to very long answers, but I also thought that it could serve as a reference to understand the relations between these functions, at least the most commonly used and useful. Eventually, people will add more answers or edit the existing ones. Also, the explanation of the relations can just be an equation (if that's sufficient). I think it would be enough, in this case, to describe the functions that are theoretically measures of an agent performance. You can ignore how they might be approximated. – nbro Feb 14 '19 at 21:11

Advantage function: $$A(s,a) = Q(s,a) - V(s)$$
More interesting is the General Value Function (GVF), the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. It is therefore a generalization of value function $$V(s)$$. The GVF is defined on page 459 of the 2nd edition of Sutton and Barto's RL book as $$v_{\pi,\gamma,C}(s) =\mathbb{E}\left[\left.\sum_{k=t}^\infty\left(\prod_{i=t+1}^k \gamma(S_i)\right)C_{k+1}\right\rvert S_t=s, A_{t:\infty}\sim\pi\right]$$ where $$C_t \in \mathbb{R}$$ is the signal being summed over time.
$$\gamma(S_t)$$ is a function $$\gamma: \cal{S}\to[0,1]$$ allowing the discount rate to depend upon the state. Sutton and Barto call it the termination function. Some call it the continuation function.
Also of note are the differential value functions. These are used in the continuing, undiscounted setting. Because there is no discounting, the expected sum of future rewards is unbounded. Instead, we optimize the expected differential reward $$R_{t+1}-r(\pi)$$, where $$r(\pi)$$ is the average reward under policy $$\pi$$.
$$v_{\pi,\,diff}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\left[r-r(\pi)+ v_{\pi,\,diff}(s')\right]$$ $$v_{*,\,diff}(s) = \max_a \sum_{s',r} p(s',r|s,a)\left[r-\max_\pi r(\pi)+ v_{*,\,diff}(s')\right]$$
The differential value functions assume that a single fixed value of $$r(\pi)$$ exists. That is, they assume the MDP is "ergodic." See section 10.3 of Sutton and Barto for details.