9
votes
Accepted
Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?
Let's first write the state-value function as
$$q_{\pi}(s,a) = \mathbb{E}_{p, \pi}[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]\;,$$
where $R_{t+1}$ is the random variable that represents the reward ...
- 4,665
8
votes
Accepted
When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?
The core differences between using $V(s)$ or $Q(s,a)$ are:
$V(s)$ cannot be used stand-alone to decide a policy.
You either need a separate policy function $\pi(a|s)$ that it is the value function for,...
- 30.2k
7
votes
Accepted
Why is the state-action value function used more than the state value function?
We are ultimately interested in getting an optimal policy, that is the optimal sequence of actions to reach the final goal. State values on its own don't provide that, they tell you expected return ...
- 2,336
7
votes
Accepted
What is the difference between expected return and value function?
There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
- 30.2k
6
votes
Accepted
How do we express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$?
Not quite. You are missing the reward at time step $t+1$.
The definition you are looking for is (leaving out the $\pi$ subscripts for ease of notation)
$$q(s,a) = \mathbb{E}[R_{t+1} + \gamma v(s') | ...
- 4,665
6
votes
In Value Iteration, why can we initialize the value function arbitrarily?
Is this something to do with the Bellman optimality constraint itself?
That is part of it, and important for episodic problems without discounting. The Bellman equations link between time steps, ...
- 30.2k
5
votes
Accepted
How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?
There seem to be two different ideas in this question here:
What's the impact / importance of our choice for reward values?
What's the impact / importance of our choice for initial value estimates (...
- 10.1k
5
votes
Accepted
Apart from the state and state-action value functions, what are other examples of value functions used in RL?
Advantage function: $A(s,a) = Q(s,a) - V(s)$
More interesting is the General Value Function (GVF), the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. ...
- 2,028
5
votes
Accepted
How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?
Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values.
The ...
- 10.1k
5
votes
Accepted
What is the difference between a greedy policy and an optimal policy?
I would like to know if the optimal value function can also be defined as
$$v_*(s_t) = \max_{a \in A(s_t)} \big\{ E_F \left[ r_{t+1} | s_t,a \right]+ \delta E_F \left[v_* \left(s_{t+1}\right)| s_t,a \...
- 30.2k
4
votes
Accepted
In reinforcement learning, does the optimal value correspond to performing the best action in a given state?
I am wondering which definition is correct.
The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy"
So this one is ...
- 30.2k
4
votes
Accepted
Why is there an expectation sign in the Bellman equation?
There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons-
The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\...
- 78
4
votes
What is the target Q-value in DQNs?
When training a Deep Q network with experienced replay, you accumulate what is known as training experiences $e_t = (s_t, a_t, r_t, s_{t+1})$. You then sample a batch of such experiences and for each ...
- 1,251
4
votes
Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?
David Ireland gives a fantastic answer, and I will provide an intuitive and gentle (but less rigorous) answer for those who are unfamiliar with the relevant statistical concepts.
Next reward $R_{t+1}...
- 1,282
4
votes
Why are the value functions sometimes written with capital letters and other times with lower-case letters?
In the Sutton and Barto book $q(s,a)$ is used to denote the true expected value of taking action $a$ in state $s$, whereas capital $Q(s,a)$ is used to denote an estimate of $q(s,a)$. However, there is ...
- 4,665
4
votes
Accepted
How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?
isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely
$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{...
- 30.2k
4
votes
Accepted
Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$
Your understanding of the Bellman equation is not quite right. The state-action value function is defined as the expected (discounted) returns when taking action $a$ in state $s$. Now, in your ...
- 4,665
4
votes
Accepted
What is a "learned policy" in Q-learning?
A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed ...
- 30.2k
4
votes
Accepted
How would I compute the optimal state-action value for a certain state and action?
It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.
Remember that to simply obtain an optimal Q-value for a given state-action ...
- 4,665
4
votes
Accepted
In Value Iteration, why can we initialize the value function arbitrarily?
If the value function of a state $v(s)$ is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit $s$, since the high $v(s)$ makes it very promising. The ...
- 1,282
4
votes
Accepted
What is the difference between these two versions of the Bellman equation?
The two are equivalent.
\begin{align}
V_\pi(s) &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]\\
&= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s'|s,a)p(r| s',a,s)[r + \gamma ...
- 580
3
votes
Accepted
Are these two definitions of the state-action value function equivalent?
The definition of the state-action value function is always the same.
Your definition is correct, as $q_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q_{\pi}(s,a)$ as an conditional ...
- 39.5k
3
votes
Accepted
Do policy independent state and action values exist in reinforcement learning?
Do policy independent state and action values exist in reinforcement learning?
No. They do not exist, because in order to progress in any MDP and receive any reward - i.e. to get any measure of value ...
- 30.2k
3
votes
What is the target Q-value in DQNs?
The deep Q-learning (DQL) algorithm is really similar to the tabular Q-learning algorithm. I think that both algorithms are actually quite simple, at least, if you look at their pseudocode, which isn'...
- 39.5k
3
votes
What is the target Q-value in DQNs?
What does the target Q-values represent?
In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action $a$ in state $s$, and from ...
- 30.2k
3
votes
Accepted
Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?
Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$?
The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...
- 30.2k
3
votes
Is there any grid world dataset or generator for reinforcement learning?
Depending on your needs and the size of the project, you might be better off making a custom set of environments. If you'd rather not do that, though, you should take a look at OpenAI's CoinRun ...
- 2,028
3
votes
Accepted
In reinforcement learning, is the value of terminal/goal state always zero?
In reinforcement learning, is the value of terminal/goal state always zero?
Yes, always for episodic problems, the value of a terminal state is always zero, from the definition.
The value of a state $...
- 30.2k
3
votes
Accepted
Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram?
Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not ...
- 30.2k
Only top scored, non community-wiki answers of a minimum length are eligible