# Questions tagged [value-functions]

For questions related to the concept of value (or performance, or quality, or utility) function (as defined in reinforcement learning and other AI sub-fields). An example of this type of functions is the Q function (used e.g. in the Q-learning algorithm), also known as the state-action value function, given that $Q: S \times A \rightarrow \mathbb{R}$, where $S$ and $A$ are respectively the set of states and actions of the environment.

46 questions
Filter by
Sorted by
Tagged with
179 views

### Why is the state-action value function used more than the state value function?

In reinforcement learning, the state-action value function seems to be used more than the state value function. Why is it so?
485 views

### What is the target Q-value in DQNs?

I understand that in DQNs, the loss is measured by taking the MSE of outputted Q-values and target Q-values. Whats does the target Q-values represent? And how is it obtained/calculated by the DQN?
167 views

### Why are the value functions sometimes written with capital letters and other times with lower-case letters?

Why are the state-value and action-value functions are sometimes written in small letters and other times in capitals? For instance, why in the Q-learning algorithm (page 131 of Barto and Sutton's ...
108 views

### Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

In equation 3.17 of Sutton and Barto's book: $$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$ $G_{t+1}$ here have been replaced with $v_*(S_{t+1})$, but no reason has ...
191 views

### An example of a unique value function which is associated with multiple optimal policies

In the 4th paragraph of http://www.incompleteideas.net/book/ebook/node37.html it is mentioned: Whereas the optimal value functions for states and state-action pairs are unique for a given MDP, ...
148 views

### What is the value of a state when there is a certain probability that agent will die after each step?

We assume infinite horizon and discount factor $\gamma = 1$. At each step, after the agent takes an action and gets its reward, there is a probability $\alpha = 0.2$, that agent will die. The assumed ...
167 views

### How do we express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$?

The task (exercise 3.13 in the RL book by Sutton and Barto) is to express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$. $q_\pi(s,a)$ is the action-value function, that states how good ...
437 views

### Apart from the state and state-action value functions, what are other examples of value functions used in RL?

In reinforcement learning, we often define two functions, the state-value function $$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$ and the state-...
92 views

Equation 7.3 of Sutton Barto book: $$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)|$$ $$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}... 1answer 95 views ### In reinforcement learning, does the optimal value correspond to performing the best action in a given state? I am confused about the definition of the optimal value (V^*) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub ... 1answer 60 views ### How are afterstate value functions mathematically defined? In this answer, afterstate value functions are mentioned, and that temporal-difference (TD) and Monte Carlo (MC) methods can also use these value functions. Mathematically, how are these value ... 1answer 56 views ### Can we stop training as soon as epsilon is small? I'm new to reinforcement learning. As it is common in RL, \epsilon-greedy search for the behavior/exploration is used. So, at the beginning of the training, \epsilon is high, and therefore a lot ... 1answer 54 views ### How is the incremental update rule derived from the weighted importance sampling in off-policy Monte Carlo control? Here's the approximated value using weighted importance sampling$$ V_{n} \doteq \frac{\sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n-1} W_{k}}, \quad n \geq 2 $$Here's the incremental update rule for ... 4answers 349 views ### How to stop DQN Q function from increasing during learning? Following the DQN algorithm with experience replay: We calculate the loss=(Q(s,a)-(r+Q(s+1,a)))^2. Assume I have positive but changing rewards. Meaning, r>0. Thus, since the rewards are ... 1answer 91 views ### Why is the state value function sufficient to determine the policy if a model is available? In section "5.2 Monte Carlo Estimation of Action Values" of the second edition of the reinforcement learning book by Sutton and Barto, this is stated: If a model is not available, then it is ... 0answers 62 views ### In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update? While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6)$$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
49 views

In chapter 3.5 of Sutton's book, the value function is defined as: Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the ...
102 views

### How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...
44 views

### Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference? Here's the update formula.
124 views

### Equation not satisfied in Policy Iteration Algorithm

In equation 4.9 of Sutton and Barto's book on page 79, we have(for policy iteration algo): $\pi ^{'}(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi}(s')]$ where $\pi$ is the previous policy ...
100 views

### Are these two definitions of the state-action value function equivalent?

I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function ...
87 views

### Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
128 views

### Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
### How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?
This is the exercise 3.18 in Sutton and Barto's book. The task is to express $v_\pi(s)$ using $q_\pi(s,a)$. Looking at the diagram above, the value of $q_\pi(s,a)$ at $s$ for each $a \in A$ we take ...