# Tag Info

Accepted

### Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?

Let's first write the state-value function as $$q_{\pi}(s,a) = \mathbb{E}_{p, \pi}[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]\;,$$ where $R_{t+1}$ is the random variable that represents the reward ...
• 4,410
Accepted

### Why is the state-action value function used more than the state value function?

We are ultimately interested in getting an optimal policy, that is the optimal sequence of actions to reach the final goal. State values on its own don't provide that, they tell you expected return ...
• 2,286
Accepted

### What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
• 26.5k
Accepted

### When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?

The core differences between using $V(s)$ or $Q(s,a)$ are: $V(s)$ cannot be used stand-alone to decide a policy. You either need a separate policy function $\pi(a|s)$ that it is the value function for,...
• 26.5k
Accepted

• 26.5k
Accepted

### In reinforcement learning, does the optimal value correspond to performing the best action in a given state?

I am wondering which definition is correct. The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy" So this one is ...
• 26.5k
Accepted

• 1,122

### Why are the value functions sometimes written with capital letters and other times with lower-case letters?

In the Sutton and Barto book $q(s,a)$ is used to denote the true expected value of taking action $a$ in state $s$, whereas capital $Q(s,a)$ is used to denote an estimate of $q(s,a)$. However, there is ...
• 4,410
Accepted

isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely $v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{... • 26.5k 4 votes Accepted ### What is a "learned policy" in Q-learning? A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed ... • 26.5k 4 votes Accepted ### How would I compute the optimal state-action value for a certain state and action? It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values. Remember that to simply obtain an optimal Q-value for a given state-action ... • 4,410 4 votes Accepted ### In Value Iteration, why can we initialize the value function arbitrarily? If the value function of a state$v(s)$is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit$s$, since the high$v(s)makes it very promising. The ... • 1,122 4 votes Accepted ### What is the difference between these two versions of the Bellman equation? The two are equivalent. \begin{align} V_\pi(s) &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s'|s,a)p(r| s',a,s)[r + \gamma ... • 560 3 votes Accepted ### Are these two definitions of the state-action value function equivalent? The definition of the state-action value function is always the same. Your definition is correct, asq_{\pi}(s,a)$is conditioned on$a$, so you don't need to write$q_{\pi}(s,a)$as an conditional ... • 37k 3 votes Accepted ### Do policy independent state and action values exist in reinforcement learning? Do policy independent state and action values exist in reinforcement learning? No. They do not exist, because in order to progress in any MDP and receive any reward - i.e. to get any measure of value ... • 26.5k 3 votes ### What is the target Q-value in DQNs? The deep Q-learning (DQL) algorithm is really similar to the tabular Q-learning algorithm. I think that both algorithms are actually quite simple, at least, if you look at their pseudocode, which isn'... • 37k 3 votes ### What is the target Q-value in DQNs? What does the target Q-values represent? In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action$a$in state$s$, and from ... • 26.5k 3 votes Accepted ### Why is$G_{t+1}$is replaced with$v_*(S_{t+1})$in the Bellman optimality equation? Can someone provide the reasoning behind why$G_{t+1}$is equal to$v_*(S_{t+1})$? The two things are not usually exactly equal, because$G_{t+1}$is a probability distribution over all possible ... • 26.5k 3 votes ### Is there any grid world dataset or generator for reinforcement learning? Depending on your needs and the size of the project, you might be better off making a custom set of environments. If you'd rather not do that, though, you should take a look at OpenAI's CoinRun ... • 2,018 3 votes Accepted ### In reinforcement learning, is the value of terminal/goal state always zero? In reinforcement learning, is the value of terminal/goal state always zero? Yes, always for episodic problems, the value of a terminal state is always zero, from the definition. The value of a state$...
• 26.5k
Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not ...