8 votes
Accepted

Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?

Let's first write the state-value function as $$q_{\pi}(s,a) = \mathbb{E}_{p, \pi}[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]\;,$$ where $R_{t+1}$ is the random variable that represents the reward ...
user avatar
7 votes
Accepted

Why is the state-action value function used more than the state value function?

We are ultimately interested in getting an optimal policy, that is the optimal sequence of actions to reach the final goal. State values on its own don't provide that, they tell you expected return ...
user avatar
  • 2,236
7 votes
Accepted

What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
user avatar
  • 24.5k
6 votes

In Value Iteration, why can we initialize the value function arbitrarily?

Is this something to do with the Bellman optimality constraint itself? That is part of it, and important for episodic problems without discounting. The Bellman equations link between time steps, ...
user avatar
  • 24.5k
5 votes
Accepted

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

There seem to be two different ideas in this question here: What's the impact / importance of our choice for reward values? What's the impact / importance of our choice for initial value estimates (...
user avatar
  • 9,459
5 votes
Accepted

Apart from the state and state-action value functions, what are other examples of value functions used in RL?

Advantage function: $A(s,a) = Q(s,a) - V(s)$ More interesting is the General Value Function (GVF), the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. ...
user avatar
5 votes
Accepted

How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values. The ...
user avatar
  • 9,459
5 votes
Accepted

How do we express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$?

Not quite. You are missing the reward at time step $t+1$. The definition you are looking for is (leaving out the $\pi$ subscripts for ease of notation) $$q(s,a) = \mathbb{E}[R_{t+1} + \gamma v(s') | ...
user avatar
5 votes
Accepted

When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?

The core differences between using $V(s)$ or $Q(s,a)$ are: $V(s)$ cannot be used stand-alone to decide a policy. You either need a separate policy function $\pi(a|s)$ that it is the value function for,...
user avatar
  • 24.5k
5 votes
Accepted

What is the difference between a greedy policy and an optimal policy?

I would like to know if the optimal value function can also be defined as $$v_*(s_t) = \max_{a \in A(s_t)} \big\{ E_F \left[ r_{t+1} | s_t,a \right]+ \delta E_F \left[v_* \left(s_{t+1}\right)| s_t,a \...
user avatar
  • 24.5k
4 votes
Accepted

Why is there an expectation sign in the Bellman equation?

There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons- The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\...
user avatar
  • 78
4 votes

What is the target Q-value in DQNs?

When training a Deep Q network with experienced replay, you accumulate what is known as training experiences $e_t = (s_t, a_t, r_t, s_{t+1})$. You then sample a batch of such experiences and for each ...
user avatar
  • 1,241
4 votes

Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?

David Ireland gives a fantastic answer, and I will provide an intuitive and gentle (but less rigorous) answer for those who are unfamiliar with the relevant statistical concepts. Next reward $R_{t+1}...
user avatar
  • 958
4 votes

Why are the value functions sometimes written with capital letters and other times with lower-case letters?

In the Sutton and Barto book $q(s,a)$ is used to denote the true expected value of taking action $a$ in state $s$, whereas capital $Q(s,a)$ is used to denote an estimate of $q(s,a)$. However, there is ...
user avatar
4 votes
Accepted

How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?

isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely $v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{...
user avatar
  • 24.5k
4 votes
Accepted

What is a "learned policy" in Q-learning?

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed ...
user avatar
  • 24.5k
4 votes
Accepted

How would I compute the optimal state-action value for a certain state and action?

It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values. Remember that to simply obtain an optimal Q-value for a given state-action ...
user avatar
4 votes
Accepted

In Value Iteration, why can we initialize the value function arbitrarily?

If the value function of a state $v(s)$ is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit $s$, since the high $v(s)$ makes it very promising. The ...
user avatar
  • 958
3 votes
Accepted

Are these two definitions of the state-action value function equivalent?

The definition of the state-action value function is always the same. Your definition is correct, as $q_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q_{\pi}(s,a)$ as an conditional ...
user avatar
  • 34.5k
3 votes
Accepted

Do policy independent state and action values exist in reinforcement learning?

Do policy independent state and action values exist in reinforcement learning? No. They do not exist, because in order to progress in any MDP and receive any reward - i.e. to get any measure of value ...
user avatar
  • 24.5k
3 votes

What is the target Q-value in DQNs?

The deep Q-learning (DQL) algorithm is really similar to the tabular Q-learning algorithm. I think that both algorithms are actually quite simple, at least, if you look at their pseudocode, which isn'...
user avatar
  • 34.5k
3 votes

What is the target Q-value in DQNs?

What does the target Q-values represent? In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action $a$ in state $s$, and from ...
user avatar
  • 24.5k
3 votes
Accepted

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$? The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...
user avatar
  • 24.5k
3 votes
Accepted

In reinforcement learning, does the optimal value correspond to performing the best action in a given state?

I am wondering which definition is correct. The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy" So this one is ...
user avatar
  • 24.5k
3 votes
Accepted

In reinforcement learning, is the value of terminal/goal state always zero?

In reinforcement learning, is the value of terminal/goal state always zero? Yes, always for episodic problems, the value of a terminal state is always zero, from the definition. The value of a state $...
user avatar
  • 24.5k
3 votes
Accepted

Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram?

Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not ...
user avatar
  • 24.5k
3 votes

What is the value of a state when there is a certain probability that agent will die after each step?

The value of a state depends on the policy that you use, so I'll make the assumption here that you're talking about value using the optimal policy. According to the optimal policy, the agent would ...
user avatar
3 votes
Accepted

Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$

Your understanding of the Bellman equation is not quite right. The state-action value function is defined as the expected (discounted) returns when taking action $a$ in state $s$. Now, in your ...
user avatar
3 votes
Accepted

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Removing the learning rate will likely yield poor convergence to the optimal policy and optimal Q-values. Note that the current policy is completely dependent on the Q-values, as we take the action ...
user avatar
  • 958
3 votes
Accepted

Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem

So, naturally, if you've observed something that contradicts the theoretical properties of Value Iteration, something's wrong, right? Well, the code you've linked, as it is, is fine. It works as ...
user avatar
  • 436

Only top scored, non community-wiki answers of a minimum length are eligible