Apart from the state and state-action value functions, what are other examples of value functions used in RL?

Question

In reinforcement learning, we often define two functions, the state-value function

$$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$

and the state-action-value function

$$Q^\pi(s,a) = \mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}\Bigg|S_t=s, A_t=a \right]$$

where $\mathbb{E}_{\pi}$ means that these functions are defined as the expectation with respect to a fixed policy $\pi$ of what is often called the return, $\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}$, where $\gamma$ is a discount factor and $R_{t+k+1}$ is the reward received from the environment (while the agent interacts with it) from time $t$ onwards.

So, both the $V$ and $Q$ functions are defined as expectations of the return (or the cumulative future discounted reward), but these expectations have different "conditions" (or are conditioned on different variables). The $V$ function is the expectation (with respect to a fixed policy $\pi$) of the return given that the current state (the state at time $t$) is $s$. The $Q$ function is the expectation (with respect to a fixed policy $\pi$) of the return conditioned on the fact that the current state the agent is in is $s$ and the action the agent takes at $s$ is $a$.

Furthermore, the Bellman optimality equation for $V^*$ (the optimal value function) can be expressed as the Bellman optimality equation for $Q^{\pi^*}$ (the optimal state-action value function associated with the optimal policy $\pi^*$) as follows

$$ V^*(s) = \max_{a \in \mathcal{A}(s)} Q^{\pi^*}(s, a) $$

This is actually shown (or proved) at page 76 of the book "Reinforcement Learning: An Introduction" (1st edition) by Andrew Barto and Richard S. Sutton.

Are there any other functions, apart from the $V$ and $Q$ functions defined above, in the RL context? If so, how are they related?

For example, I've heard of the "advantage" or "continuation" functions. How are these functions related to the $V$ and $Q$ functions? When should one be used as opposed to the other? Note that I'm not just asking about the "advantage" or "continuation" functions, but, if possible, any existing function that is used in RL that is similar (in purpose) to these mentioned functions, and how they are related to each other.

Answer 1

Advantage function: $A(s,a) = Q(s,a) - V(s)$

More interesting is the General Value Function (GVF), the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. It is therefore a generalization of value function $V(s)$. The GVF is defined on page 459 of the 2nd edition of Sutton and Barto's RL book as $$v_{\pi,\gamma,C}(s) =\mathbb{E}\left[\left.\sum_{k=t}^\infty\left(\prod_{i=t+1}^k \gamma(S_i)\right)C_{k+1}\right\rvert S_t=s, A_{t:\infty}\sim\pi\right]$$ where $C_t \in \mathbb{R}$ is the signal being summed over time.

$\gamma(S_t)$ is a function $\gamma: \cal{S}\to[0,1]$ allowing the discount rate to depend upon the state. Sutton and Barto call it the termination function. Some call it the continuation function.

Also of note are the differential value functions. These are used in the continuing, undiscounted setting. Because there is no discounting, the expected sum of future rewards is unbounded. Instead, we optimize the expected differential reward $R_{t+1}-r(\pi)$, where $r(\pi)$ is the average reward under policy $\pi$.

$$v_{\pi,\,diff}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\left[r-r(\pi)+ v_{\pi,\,diff}(s')\right]$$ $$v_{*,\,diff}(s) = \max_a \sum_{s',r} p(s',r|s,a)\left[r-\max_\pi r(\pi)+ v_{*,\,diff}(s')\right]$$

The differential value functions assume that a single fixed value of $r(\pi)$ exists. That is, they assume the MDP is "ergodic." See section 10.3 of Sutton and Barto for details.

Answer 2

There is also the simpler action value function $$q_*(a) = \mathbb{E} \left[ R_t \mid A_t = a\right],$$ which we try to approximate when solving context-free bandit problems. You can also similarly define the action value function for contextual bandit problems by also conditioning on the context (rather than just on the action).

See chapter 2 of the book Reinforcement Learning: An Introduction (2nd edition) by Barto and Sutton for more details.

There are also the afterstate value functions, $v(s')$. Check this question and section 6.8 of the just cited book.

Moreover, there's the state-action-goal value function, $q(s, a, g)$, as described in the paper Hindsight Experience Replay (2017), although I am not sure how we can mathematically define it (i.e. as a function of other value functions or as a Bellman equation).