New answers tagged bellman-equations
2
It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.
Remember that to simply obtain an optimal Q-value for a given state-action pair we can evaluate
$$Q(s, a) = r + \gamma \max_{a'} Q(s', a)\;;$$
where $s'$ is the state we transitioned into (note that this only holds when obtaining the ...
3
Your equations all look correct to me.
It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy.
There are ways of working with MDPs, through sampling of actions, state transitions and rewards, where it is possible to estimate value functions without knowing either $\pi(a|s)$ or $P^{a}_{ss'}$. For instance,...
Top 50 recent answers are included
Related Tags
bellman-equations × 29reinforcement-learning × 27
value-functions × 13
proofs × 6
comparison × 4
q-learning × 4
reference-request × 3
policy-iteration × 3
expectation × 3
bellman-operators × 3
definitions × 2
convergence × 2
policies × 2
reward-functions × 2
reward-shaping × 2
policy-evaluation × 2
optimal-policy × 2
dqn × 1
terminology × 1
loss-functions × 1
math × 1
policy-gradients × 1
markov-decision-process × 1
implementation × 1
monte-carlo-methods × 1