All Questions
Tagged with comparison value-functions
15 questions
0
votes
2
answers
55
views
Is there any actual difference between these 2 definitions of a state value function?
The definition of the value function in TRPO paper is
\begin{align}
V_\pi(s_t) &= \mathbb{E}_{a_t,s_{t+1},\ldots} \left[ \sum_{l=0}^{\infty} \gamma^l r(s_{t+l}) \right], \\[10pt]
a_t &\sim \pi(...
- 139
1
vote
1
answer
96
views
How are these two equations for the optimal state-value function equivalent?
By substituting the optimal policy $\pi_{\star}$ into the Bellman equation, we get the Bellman equation for $v_{\pi_{\star}}(s)=v_{\star}(s)$:
$$ v_{\star}(s) = \sum\limits_a \pi_{\star}(a|s) \sum\...
- 1,115
3
votes
1
answer
2k
views
What is the difference between a greedy policy and an optimal policy?
I am struggling to understand what is the difference between an optimal policy and a greedy policy.
Let $F(r_{t+1},s_{t+1}| s_t,a_t)$ be the probability distribution accorting to which, given action $...
- 33
6
votes
1
answer
2k
views
When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?
I saw the difference between value function $V(s)$ and $Q(s, a)$. But when do I use each one? When I coded in Matlab I only used $Q(s, a)$ directly (as I was thinking of a tabular approach). So, when ...
6
votes
2
answers
8k
views
When to use Value Iteration vs. Policy Iteration
Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms. However, they differ in the mechanics of their updates. Policy Iteration seeks to first find a completed ...
- 541
1
vote
1
answer
1k
views
Is there any difference between an objective function and a value function?
I found the usage of both objective function and value function in the same context.
Context #1: In the paper titled Generative Adversarial Nets by Ian J. Goodfellow et al.
We simultaneously train G ...
- 3,990
0
votes
1
answer
91
views
Are the state-action values and the state value function equivalent for a given policy?
Are the state-action values and the state value function equivalent for a given policy? I would assume so as the value function is defined as $V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$. If we are operating a ...
3
votes
1
answer
1k
views
How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?
This is exercise 3.18 in Sutton and Barto's book.
The task is to express $v_\pi(s)$ using $q_\pi(s,a)$.
Looking at the diagram above, the value of $q_\pi(s,a)$ at $s$ for each $a \in A$ we take gives ...
- 392
2
votes
1
answer
269
views
Are these two definitions of the state-action value function equivalent?
I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function ...
- 5,030
0
votes
1
answer
1k
views
What is the difference between the state transition of an MDP and an action-value?
Let's say we have MDP where we have a state transition matrix.
How is this state transition different from action value in reinforcement learning? Is the state transition in MDP stochastic ...
- 171
1
vote
2
answers
3k
views
What is the relationship between the reward function and the value function?
To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
- 633
1
vote
1
answer
444
views
What is the relationship between the Q and V functions?
Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$.
Can we say that $Q^\pi(s, a) = V^\pi(s)$?
The reason why I think this can be the case is ...
- 1,291
7
votes
1
answer
935
views
Why is the state-action value function used more than the state value function?
In reinforcement learning, the state-action value function seems to be used more than the state value function. Why is it so?
- 404
3
votes
2
answers
861
views
Apart from the state and state-action value functions, what are other examples of value functions used in RL?
In reinforcement learning, we often define two functions, the state-value function
$$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty}
\gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$
and the state-...
- 41.4k
10
votes
1
answer
4k
views
What is the difference between expected return and value function?
I've seen numerous mathematical explanations of reward, value functions $V(s)$, and return functions. The reward provides an immediate return for being in a specific state. The better the reward, the ...
- 221