Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [value-functions]

For questions related to the concept of value (or performance, or quality, or utility) function (as defined in reinforcement learning and other AI sub-fields). An example of this type of functions is the Q function (used e.g. in the Q-learning algorithm), also known as the state-action value function, given that $Q: S \times A \rightarrow \mathbb{R}$, where $S$ and $A$ are respectively the set of states and actions of the environment.

Filter by
Sorted by
Tagged with
0 votes
2 answers
26 views

In reinforcement learning policy evaluation, which reward is received?

Assume the agent is in a state $s$ and takes action $a$, intending to move to $s_1$, however the dynamics may drop the agent in $s_1$ or $s_2$: may move to $s_1$ and receive reward $r_1$ with a ...
Nat's user avatar
  • 13
0 votes
1 answer
84 views

Can Q(s,a) be replaced by V(s) when certrain requirements are met? [closed]

I read this post, was thinking about it and now I have a hypothesis but I am not sure whether or not its correct. I claim that in Q-learning $Q(s,a)$ can be replaced by V(s) when $p(s'|a,s)$ is ...
NMO's user avatar
  • 161
1 vote
1 answer
53 views

How to derive "value iteration" from "policy iteration"?

This is the equation for "value iteration" from Sutton-Barto: \begin{align} v_{k+1}(s)=& \max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_k(S_{t+1}) \big|S_t=s, A_t=a\Big) \\ =&...
DSPinfinity's user avatar
2 votes
1 answer
66 views

Is $s_0$ the current state in policy gradients?

As far as I understand from here (source: OpenAI), the objective function in Policy Gradient is: $$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$ where $R(\tau)=r_0+r_1+...+r_T$, with $r_t$ ...
fermented_bean's user avatar
1 vote
0 answers
66 views

When can we unnest the minimizations/recursions in an value function(bellman optimality equation)?

When reading the following paper(page 4): An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control I could see that they were able to unnest the minimization's in the ...
richard baws's user avatar
0 votes
0 answers
18 views

Proof of Difference in Return Between Two Policies

I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
Nikhil Sridhar's user avatar
0 votes
0 answers
60 views

How are Target Values Generated in Alpha Zero Architecture

I am a little confused as to how the target values are generated to train the neural network with the Alpha Zero architecture(in specific to a chess game). I understand how the improved policy is ...
Kiran Manicka's user avatar
3 votes
2 answers
278 views

How do we get the optimal value-function?

In here it says that: (is it correct?) $$V^\pi = \sum_{a \in A}\pi(a|s)*Q^\pi(s,a)$$ And we have: $$ V^*(s) = max_\pi V^\pi(s)$$ Also: $$ V^*(s) = max_a Q^*(s, a) $$ Can someone demonstrate to me step ...
Ness's user avatar
  • 206
1 vote
0 answers
41 views

What is the definition of Q when the discount factor depends on time?

Suppose I want to find out the Q value of a particular state $s$ bu doing action $a$ at a particular timestep $t$. I know that the Q-value when the discount factor is given by, $$Q(a,s)=E_{\pi}\big[...
Asher2211's user avatar
  • 111
4 votes
1 answer
93 views

Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?

I am reading the Sutton & Barto (2018) RL textbook. On page 59, it derives the recursive property of the state-value function as below. Could you explain the steps of third and fourth equality? ...
tesio's user avatar
  • 205
0 votes
1 answer
193 views

How to prove that $V^\star$ is optimal if and only if it satisfies the Bellman equation?

The Question I'd like to prove that a function $V$ (like in reinforcement learning) is optimal iff it satisfies the bellman equation. A lot of places online reference this fact, but none prove it. ...
snatchysquid's user avatar
0 votes
1 answer
41 views

Query regarding the minimax value function of GANs

In the book Generative AI with Python and TensorFlow 2 from Babcock and Bali (page 172), it is stated that the value function of a GAN is the following: where D(x) is the output of the discriminator ...
Claudia P's user avatar
1 vote
1 answer
85 views

How are these two equations for the optimal state-value function equivalent?

By substituting the optimal policy $\pi_{\star}$ into the Bellman equation, we get the Bellman equation for $v_{\pi_{\star}}(s)=v_{\star}(s)$: $$ v_{\star}(s) = \sum\limits_a \pi_{\star}(a|s) \sum\...
DSPinfinity's user avatar
0 votes
1 answer
227 views

Will my Q values keep going up forever?

In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes,...
Kyotiq's user avatar
  • 1
4 votes
1 answer
470 views

Why are policy gradient methods more effective in high-dimensional action spaces?

David Silver argues, in his Reinforcement Learning course, that policy-based reinforcement learning (RL) is more effective than value-based RL in high-dimensional action spaces. He points out that the ...
Saucy Goat's user avatar
2 votes
1 answer
184 views

What is the difference between these two versions of the Bellman equation?

The first version is the one I am most familiar with: $$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V_\pi(s')]$$ where $T(s, a, s')$ represents the probability of ...
Nova's user avatar
  • 133
1 vote
0 answers
28 views

Proof that the Policy Iteration Converges?

Let $\mathcal{X}=:\{x_1, x_2, x_3,...,x_n\}$ be the state space. Let $\mathcal{U}:=\{u_1, u_2, u_3,...,u_m\}$ be the set of actions. Let $A^{u_1}, A^{u_2}, A^{u_3},...,A^{u_m}$ be the state transition ...
SHASHANK RANJAN's user avatar
0 votes
1 answer
210 views

Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument ...
user's user avatar
  • 145
0 votes
0 answers
81 views

Policy and Discount Factor

This question is similar to this question, however it has a different question. I'm learning MDP's and I'd like to know if I'm doing these exercises correctly: Consider the following MDP: Suppose a ...
user's user avatar
  • 145
0 votes
0 answers
35 views

Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...
user's user avatar
  • 145
1 vote
2 answers
608 views

Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?

In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state: $$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)} $$ In Q-learning, we ...
Cloudy's user avatar
  • 223
4 votes
2 answers
393 views

How do we estimate the value of a stochastic policy?

I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
mac_or_cheese's user avatar
7 votes
2 answers
2k views

In Value Iteration, why can we initialize the value function arbitrarily?

I have not been able to find a good explanation of this, other than statements that the algorithm is guaranteed to converge with arbitrary choices for initial values in each state. Is this something ...
Arham's user avatar
  • 73
1 vote
1 answer
638 views

PPO custom implementation: do metrics like value loss, actor loss and entropy move a certain way?

I'm wondering whether problems with a custom PPO implementation (problem couldn't be made into a gym environment) can be debugged considering how the losses change over time. In my current experiment, ...
Antonis Karvelas's user avatar
3 votes
1 answer
97 views

Can we also estimate $V_{\pi}$ with SARSA?

For SARSA, I know we can estimate the action value $Q(s,a)$, and the relationship between $V(s)$ and $Q(s,a)$ is $V_{\pi}(s) = \sum_{a \in \mathcal{A}} \pi(a|s)Q_{\pi} (s,a)$. So my question is, can ...
Dingzhi Hu's user avatar
1 vote
0 answers
42 views

Does all GAN's in literature need to satisfy the properties of objective function of initial GAN? [closed]

Consider the following value function of the initial GAN $V(D, G) = \mathbb{E}_{x \sim p_{data(x)}} [\log D(x)] + \mathbb{E}_{z \sim p_z(z)} [1- \log D(G(z))]$ The min-max game on the value function: $...
hanugm's user avatar
  • 3,880
0 votes
1 answer
133 views

How is policy iteration capable of improving on a deterministic policy?

Given a policy $\pi$ and the improved version upon it using policy iteration $\pi'$ we have, for $\forall s \in S$, $v_{\pi'}(s)\geq v_{\pi}(s)$. I think the way we choose $\pi'$ makes it ...
Daviiid's user avatar
  • 575
3 votes
1 answer
2k views

What is the difference between a greedy policy and an optimal policy?

I am struggling to understand what is the difference between an optimal policy and a greedy policy. Let $F(r_{t+1},s_{t+1}| s_t,a_t)$ be the probability distribution accorting to which, given action $...
fennel's user avatar
  • 33
2 votes
1 answer
204 views

Are these two forms of the state value function the same?

Why are there different forms of the value function in reinforcement learning? Sutton & Barto (2nd edition, equation 3.14) define the state value function as follows $$v_{\pi}(s) = \displaystyle\...
cgo's user avatar
  • 185
2 votes
1 answer
477 views

calculating the value of a state in an optimal policy analytically and iteratively

I am watching the lecture by Abbeel on MDPs and Reinforcement Learning. The setup of the problem is the classic gridworld with optimal policy (and corresponding values of states) pictured below. The ...
cgo's user avatar
  • 185
2 votes
1 answer
366 views

Is the initialisation of $V(s)$ and $\pi(s)$ really arbitrary in policy iteration?

In Sutton and Barto's book (Reinforcement learning: An introduction. MIT press, 2018), the algorithm "Policy Iteration" is: Here, $V(s)$ is initialized arbitrarily, meaning that I can ...
Gregwar's user avatar
  • 121
6 votes
1 answer
2k views

When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?

I saw the difference between value function $V(s)$ and $Q(s, a)$. But when do I use each one? When I coded in Matlab I only used $Q(s, a)$ directly (as I was thinking of a tabular approach). So, when ...
knowledge_seeker's user avatar
0 votes
0 answers
146 views

What do we actually 'approximate' when dealing with large state spaces in Q-learning?

I realized that my state space is very large in size. I had planned to use tabular Q-learning (Bellman equation to update the $Q(s, a)$ after each action taken). But this 'large space' realization has ...
knowledge_seeker's user avatar
5 votes
1 answer
194 views

In TD(0) with linear function approximation, why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ wrt parameters $\mathbf w$ not considered?

I am reading these slides. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this. The cost function (RMSE) is given on page ...
A Yoghes's user avatar
3 votes
1 answer
430 views

Can an optimal policy have a value function that has a smaller value for a state than a non-optimal policy?

I'm starting to learn about the Bellman Equation and a question came to my mind. A policy $\pi$ is optimal if the value $v_\pi(s)$ is greater or equal than the value $v_{\pi'}(s)$ for all states $s \...
raphael_mav's user avatar
6 votes
2 answers
7k views

When to use Value Iteration vs. Policy Iteration

Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms. However, they differ in the mechanics of their updates. Policy Iteration seeks to first find a completed ...
SeeDerekEngineer's user avatar
0 votes
1 answer
66 views

Discard irrelavant states from a MDP

I came across this question about MDP. From the look of it, it seems the full MDP is reducible if the discarded state only have 1 way in and out but is it really so if we change the discounted factor? ...
Kronic's user avatar
  • 49
2 votes
2 answers
3k views

PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
hridayns's user avatar
  • 233
1 vote
1 answer
86 views

What is the intuition behind comparing action values to state values in the policy improvement theorem?

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$. What is ...
bonzo_pippinpaddle's user avatar
3 votes
2 answers
396 views

Why must the value of a state under an optimal policy equal the expected return for the best action from that state?

The Sutton and Barto reinforcement learning textbook states that the value of a state under an optimal policy must equal the expected return for the best action from that state. That is, $$v_*(s) = \...
bonzo_pippinpaddle's user avatar
0 votes
0 answers
145 views

What are the recurrences used for updating state value function in $TD$ and $TD(\lambda)$ learning?

There are two types of value functions in reinforcement learning: State value function $V^{\pi} (s)$, state-action value function $Q^{\pi}(s, a)$. State value function: This value tells us how good ...
hanugm's user avatar
  • 3,880
1 vote
1 answer
1k views

Is there any difference between an objective function and a value function?

I found the usage of both objective function and value function in the same context. Context #1: In the paper titled Generative Adversarial Nets by Ian J. Goodfellow et al. We simultaneously train G ...
hanugm's user avatar
  • 3,880
3 votes
2 answers
1k views

How can we find the value function by solving a system of linear equations?

I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
phil's user avatar
  • 143
1 vote
1 answer
325 views

Is it possible to have values of the states equal to $0$ at the end of the value iteration?

I am new to Reinforcement Learning and I am trying to self learn it. I have already posted some quesiton here and your answershave been really useful to me, so here I am posting another one. I am ...
dcr's user avatar
  • 57
4 votes
1 answer
153 views

How to prove the second form of Bellman's equation?

I'd like to prove this "second form" of Bellman's equation: $v(s) = \mathbb{E}[R_{t + 1} + \gamma v(S_{t+1}) \mid S_{t} = s]$ starting from Bellman's equation: $v(s) = \mathbb{E}[G_{t} \mid ...
Daviiid's user avatar
  • 575
2 votes
1 answer
127 views

Is the existence and uniqueness of the state-value function for $\gamma < 1$ theoretical?

Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book. The existence and uniqueness of $V^{\pi}$ are guaranteed as long as either $\gamma < 1$...
hanugm's user avatar
  • 3,880
1 vote
0 answers
76 views

Is there any thumb rule on the cardinality of state space in order to use the parameterized function to estimate value functions?

Value functions for a given MDP can be learned in at least two ways by experience. The first way (tabular calculation) is generally used in the case of state spaces that are small enough. The second ...
hanugm's user avatar
  • 3,880
4 votes
1 answer
292 views

How is the state-value function expressed as a product of sums?

The state-value function for a given policy $\pi$ is given by $$\begin{align} V^{\pi}(s) &=E_{\pi}\left\{r_{t+1}+\gamma r_{t+2}+\gamma^{2} r_{t+3}+\cdots \mid s_{t}=s\right\} \\ &=E_{\pi}\...
hanugm's user avatar
  • 3,880
0 votes
1 answer
152 views

Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?

Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots, s_n]$. Furthermore, assume the MDP is fully observable and time is discrete. Is ...
Sbean's user avatar
  • 1
1 vote
1 answer
647 views

Bellman Expectation Equation leading to results where value iteration would not converge to the optimal policy

When applying the bellman expectation equation: $$v(s)=\mathbb{E}\left[R_{t+1}+\gamma v\left(S_{t+1}\right) \mid S_{t}=s\right]$$ to the MRP below, states further away from the terminal state will ...
Paternostro's user avatar