Questions tagged [value-functions]

For questions related to the concept of value (or performance, or quality, or utility) function (as defined in reinforcement learning and other AI sub-fields). An example of this type of functions is the Q function (used e.g. in the Q-learning algorithm), also known as the state-action value function, given that $Q: S \times A \rightarrow \mathbb{R}$, where $S$ and $A$ are respectively the set of states and actions of the environment.

Filter by
Sorted by
Tagged with
3
votes
2answers
436 views

Apart from the state and state-action value functions, what are other examples of value functions used in RL?

In reinforcement learning, we often define two functions, the state-value function $$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$ and the state-...
3
votes
1answer
91 views

Why is the state value function sufficient to determine the policy if a model is available?

In section "5.2 Monte Carlo Estimation of Action Values" of the second edition of the reinforcement learning book by Sutton and Barto, this is stated: If a model is not available, then it is ...
3
votes
1answer
57 views

How are afterstate value functions mathematically defined?

In this answer, afterstate value functions are mentioned, and that temporal-difference (TD) and Monte Carlo (MC) methods can also use these value functions. Mathematically, how are these value ...
2
votes
1answer
87 views

Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...