3 votes

Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?

There is a difference between accurate value function estimates, and optimal value functions. An optimal value function is more specifically the value function of an optimal policy. Value functions ...
user avatar
  • 24.5k
3 votes
Accepted

Is value iteration stopped after one update of each state?

Where the author mentions the policy evaluation being stopped after one state, they are referring to the part of the algorithm that evaluates the policy -- the pseudocode you have listed is the ...
user avatar
2 votes
Accepted

How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?

I don't understand how did we get rid of the condition $A_{t}=\pi'(s)$. We don't really, it is just moved into the subscript $\pi'$ in $\mathbb{E}_{\pi'}[]$ - it means the same thing here, that the ...
user avatar
  • 24.5k
2 votes
Accepted

Understanding the update rule for the policy in the policy iteration algorithm

A policy can be stochastic or deterministic. A deterministic policy is a function of the form $\pi_{\text{deterministic}}: S \rightarrow A$, that is, a function from the set of states to the set of ...
user avatar
  • 34.5k
1 vote
Accepted

Monte Carlo epsilon-greedy Policy Iteration: monotonic improvement for all cases or for the expected value?

I think this equation answer your question: $$ q_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum_{a \in A}\pi^{i+1}(a|s)q_{\pi^{i}}(s,a)$$ value of the Q while taking ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible