What's the deal with Deno? We talk with a major contributor to find out. Listen now.

Questions tagged [value-function]

For questions related to the concept of value (or performance, or quality, or utility) function (as defined in reinforcement learning and other AI sub-fields). An example of this type of functions is the Q function (used e.g. in the Q-learning algorithm).

Filter by
Sorted by
Tagged with
2
votes
1answer
75 views

What is wrong with equation 7.3 in Sutton & Barto's book?

Equation 7.3 of Sutton Barto book: $$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)| $$ $$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}...
1
vote
1answer
85 views

Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem

In Example 4.3:Gambler's Problem of Sutton and Barto's book whose code is given here. In this code the value function array is initialized as np.zeros(states) where ...
2
votes
0answers
31 views

What's an example of a simple policy but a complex value function?

Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
1
vote
1answer
37 views

The are some fundamental learning theories for developing an AI that imitates human behavior

Most if not all AI systems are not to imitate human, but to finally out-perform human. Examples include using AI to play a game, classification problems, auto-driving, and goal-oriented chatbots. ...
2
votes
1answer
96 views

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
2
votes
1answer
44 views

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference? Here's the update formula.
2
votes
1answer
39 views

How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?

This is the exercise 3.18 in Sutton and Barto's book. The task is to express $v_\pi(s)$ using $q_\pi(s,a)$. Looking at the diagram above, the value of $q_\pi(s,a)$ at $s$ for each $a \in A$ we take ...
0
votes
1answer
63 views

How do I know that the DQN has learnt an appropriate Q function?

Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it ...
3
votes
2answers
145 views

What is the value of a state when there is a certain probability that agent will die after each step?

We assume infinite horizon and discount factor $\gamma = 1$. At each step, after the agent takes an action and gets its reward, there is a probability $\alpha = 0.2$, that agent will die. The assumed ...
4
votes
2answers
163 views

Why are the value functions sometimes written with capital letters and other times with lower-case letters?

Why are the state-value and action-value functions are sometimes written in small letters and other times in capitals? For instance, why in the Q-learning algorithm (page 131 of Barto and Sutton's ...
3
votes
1answer
146 views

How do we express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$?

The task (exercise 3.13 in the RL book by Sutton and Barto) is to express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$. $q_\pi(s,a)$ is the action-value function, that states how good ...
2
votes
1answer
107 views

Equation not satisfied in Policy Iteration Algorithm

In equation 4.9 of Sutton and Barto's book on page 79, we have(for policy iteration algo): $\pi ^{'}(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi}(s')]$ where $\pi$ is the previous policy ...
4
votes
2answers
107 views

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

In equation 3.17 of Sutton and Barto's book: $$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$ $G_{t+1}$ here have been replaced with $v_*(S_{t+1})$, but no reason has ...
3
votes
1answer
51 views

How is the incremental update rule derived from the weighted importance sampling in off-policy Monte Carlo control?

Here's the approximated value using weighted importance sampling $$ V_{n} \doteq \frac{\sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n-1} W_{k}}, \quad n \geq 2 $$ Here's the incremental update rule for ...
2
votes
1answer
92 views

Are these two definitions of the state-action value function equivalent?

I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function ...
0
votes
1answer
35 views

What is the difference between the state transition of an MDP and an action-value?

Let's say we have MDP where we have a state transition matrix. How is this state transition different from action value in reinforcement learning? Is the state transition in MDP stochastic ...
2
votes
1answer
41 views

Do policy independent state and action values exist in reinforcement learning?

The state value function $V(s)$ is defined as the expected return starting in state $s$ and acting according to the current policy $\pi(a|s)$ till the end of the episode. The state-action values $Q(s,...
4
votes
3answers
364 views

What is the target Q-value in DQNs?

I understand that in DQNs, the loss is measured by taking the MSE of outputted Q-values and target Q-values. Whats does the target Q-values represent? And how is it obtained/calculated by the DQN?
2
votes
1answer
62 views

Why do I need an initial arbitrary policy to implement value iteration algorithm

I've been recently given an assignment based on Reinforcement Learning and I'm supposed to implement the value iteration algorithm in a grid environment. The assignment: My doubt is why do I even ...
1
vote
2answers
45 views

Why does the policy $\pi$ affect the Q value?

From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state). However, I am confused about the Q value and how it is "affected" by a policy. ...
1
vote
2answers
79 views

Is the Q value the same as the state-action pair value?

Am I right to say that the Q value of a particular state and action is the same as the state-action pair value of that same state and action?
1
vote
1answer
52 views

What is the relationship between the reward function and the value function?

To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
2
votes
2answers
48 views

Why is there an expectation sign in the Bellman equation?

In chapter 3.5 of Sutton's book, the value function is defined as: Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the ...
1
vote
1answer
63 views

What is the relationship between the Q and V functions?

Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$. Can we say that $Q^\pi(s, a) = V^\pi(s)$? The reason why I think this can be the case is ...
0
votes
0answers
44 views

Understanding V- and Q-functions

Assume the existence of a Markov Decision Process consisting of: State space $S$ Action space $A$ Transition model $T: S \times A \times S \to [0,1]$ Reward function $R: S \times A \times S \to \...
0
votes
1answer
36 views

What is the purpose of the arrow $\leftarrow$ in this formula?

What is the purpose of the arrow $\leftarrow$ in the formula below? $$V(S_t) \leftarrow V(S_t) + \alpha \left[ G_t - V(S_t) \right]$$ I presume it's not the same as 'equals'.
5
votes
1answer
156 views

Why is the state-action value function used more than the state value function?

In reinforcement learning, the state-action value function seems to be used more than the state value function. Why is it so?
1
vote
1answer
38 views

Value-based methods for stochastic policies

Is it possible for value-based methods to learn stochastic policies? I'm trying to get a clear picture of the different categories for RL algorithms, and while doing so I started to think about ...
2
votes
1answer
317 views

Q Learning for FrozenLake environment not converging to V* values from Value Iteration

I am trying to learn tabular Q learning, value iteration using the classical algorithms (no neural networks) by using a table of states and actions. I was trying it out on FrozenLake environment in ...
2
votes
4answers
283 views

How to stop DQN Q function from increasing during learning?

Following the DQN algorithm with experience replay: We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$. Assume I have positive but changing rewards. Meaning, $r>0$. Thus, since the rewards are ...
2
votes
1answer
72 views

Is there any grid world dataset or generator for reinforcement learning?

I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...
2
votes
1answer
72 views

Why is the state value function sufficient to determine the policy if a model is available?

In section "5.2 Monte Carlo Estimation of Action Values" of the second edition of the reinforcement learning book by Sutton and Barto, this is stated: If a model is not available, then it is ...
3
votes
1answer
365 views

What are the value functions used in reinforcement learning?

In reinforcement learning, we often define two functions, the state-value function $$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$ and the state-...
4
votes
1answer
186 views

An example of a unique value function which is associated with multiple optimal policies

In the 4th paragraph of http://www.incompleteideas.net/book/ebook/node37.html it is mentioned: Whereas the optimal value functions for states and state-action pairs are unique for a given MDP, ...