# Tag Info

## Hot answers tagged value-functions

Accepted

### Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?

Let's first write the state-value function as $$q_{\pi}(s,a) = \mathbb{E}_{p, \pi}[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]\;,$$ where $R_{t+1}$ is the random variable that represents the reward ...
• 4,920
Accepted

### When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?

The core differences between using $V(s)$ or $Q(s,a)$ are: $V(s)$ cannot be used stand-alone to decide a policy. You either need a separate policy function $\pi(a|s)$ that it is the value function for,...
• 32.6k
Accepted

### Why is the state-action value function used more than the state value function?

We are ultimately interested in getting an optimal policy, that is the optimal sequence of actions to reach the final goal. State values on its own don't provide that, they tell you expected return ...
• 2,406
Accepted

### What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
• 32.6k
Accepted

• 32.6k
Accepted

### In reinforcement learning, does the optimal value correspond to performing the best action in a given state?

I am wondering which definition is correct. The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy" So this one is ...
• 32.6k
Accepted

• 1,399

### Why are the value functions sometimes written with capital letters and other times with lower-case letters?

In the Sutton and Barto book $q(s,a)$ is used to denote the true expected value of taking action $a$ in state $s$, whereas capital $Q(s,a)$ is used to denote an estimate of $q(s,a)$. However, there is ...
• 4,920
Accepted

### What is a "learned policy" in Q-learning?

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed ...
• 32.6k
Accepted

### How would I compute the optimal state-action value for a certain state and action?

It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values. Remember that to simply obtain an optimal Q-value for a given state-action ...
• 4,920

### Are these two forms of the state value function the same?

There are a few different, but equivalent, ways to express the relationships between value functions in the Bellman equations. Some differences are just notation, but in the case of the two equations ...
• 32.6k
Accepted

### In Value Iteration, why can we initialize the value function arbitrarily?

If the value function of a state $v(s)$ is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit $s$, since the high $v(s)$ makes it very promising. The ...
• 1,399
Accepted

### What is the difference between these two versions of the Bellman equation?

The two are equivalent. \begin{align} V_\pi(s) &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s'|s,a)p(r| s',a,s)[r + \gamma ...
• 579
Accepted

### Are these two definitions of the state-action value function equivalent?

The definition of the state-action value function is always the same. Your definition is correct, as $q_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q_{\pi}(s,a)$ as an conditional ...
• 40.9k
Accepted

### Do policy independent state and action values exist in reinforcement learning?

Do policy independent state and action values exist in reinforcement learning? No. They do not exist, because in order to progress in any MDP and receive any reward - i.e. to get any measure of value ...
• 32.6k

### What is the target Q-value in DQNs?

The deep Q-learning (DQL) algorithm is really similar to the tabular Q-learning algorithm. I think that both algorithms are actually quite simple, at least, if you look at their pseudocode, which isn'...
• 40.9k

### What is the target Q-value in DQNs?

What does the target Q-values represent? In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action $a$ in state $s$, and from ...
• 32.6k

### Is there any grid world dataset or generator for reinforcement learning?

Depending on your needs and the size of the project, you might be better off making a custom set of environments. If you'd rather not do that, though, you should take a look at OpenAI's CoinRun ...
• 2,056