# Tag Info

Accepted

• 32.1k
Accepted

### In reinforcement learning, does the optimal value correspond to performing the best action in a given state?

I am wondering which definition is correct. The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy" So this one is ...
• 32.1k

• 78
Accepted

### Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$

Your understanding of the Bellman equation is not quite right. The state-action value function is defined as the expected (discounted) returns when taking action $a$ in state $s$. Now, in your ...
• 4,900
Accepted

### Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?

You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations. Policy iteration In policy iteration, a policy lookup table ...
• 32.1k
Accepted

### How would I compute the optimal state-action value for a certain state and action?

It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values. Remember that to simply obtain an optimal Q-value for a given state-action ...
• 4,900
Accepted

### What is the difference between these two versions of the Bellman equation?

The two are equivalent. \begin{align} V_\pi(s) &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]\\ &= \sum_{a}^{}\pi(a|s) \sum_{s',r}^{}p(s'|s,a)p(r| s',a,s)[r + \gamma ...
• 579

### Bellman equation and inverse matrix method

In reinforcement learning, the apostrophe character $(')$ appended to a signal usually represents the signal at the next time step. For example, $s'$ is the state in the time step immediately after $s$...
• 1,379
Accepted

### How can we find the value function by solving a system of linear equations without knowing the policy?

Your equations all look correct to me. It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy. There are ways of working with MDPs, through ...
• 32.1k
Accepted

### How are afterstate value functions mathematically defined?

Based on this and this resources, let me give an answer to my own question, but, essentially, I will just rewrite the contents of the first resource here, for reproducibility, with some minor changes ...
• 40.6k
Accepted

### Why we don't use importance sampling in tabular Q-Learning?

In Tabular Q-learning the update is as follows $$Q(s,a) = Q(s,a) + \alpha \left[R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a) \right]\;.$$ Now, as we are interested in learning about the optimal policy, ...
• 4,900
Accepted

### Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$? The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...
• 32.1k
Accepted

### Are these two definitions of the state-action value function equivalent?

The definition of the state-action value function is always the same. Your definition is correct, as $q_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q_{\pi}(s,a)$ as an conditional ...
• 40.6k
Accepted

### What do the terms 'Bellman backup' and 'Bellman error' mean?

A Bellman backup is an application of a Bellman operator. For example, the step $$V(x)\leftarrow \alpha(R + \mathbf{E}[V(x')]) + (1-\alpha)V(x)$$ Is a Bellman backup for some learning rate $\alpha$. ...
• 1,136

### How can we find the value function by solving a system of linear equations?

Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from ...
• 32.1k
Accepted

• 40.6k

### What is the Bellman equation for V(s) in the case of a deterministic environment?

You're correct, that's the definition of the Bellman equation in the deterministic case. You can refer to the Wikipedia article of the Bellman equation where $F(x, a)$ is the reward function, with $x$ ...
Accepted

### Determining to terminate at a reward or not

I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number 3 or to continue to the state with a number 4 to collect the more reward? ...
• 32.1k

### Can we also estimate $V_{\pi}$ with SARSA?

What you suggest will work, the main restriction is needing to know $\pi$ fully in order to perform the conversion. If you know that you are going to be estimating $V_{\pi}$ from the start, and have a ...
• 32.1k
Your first equation is the definition of any state value function, so it must also be definition of the optimal state value function $v_*$. The second equation is the definition of $v_*$ in terms of ...