# Tag Info

### Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?

There is a difference between accurate value function estimates, and optimal value functions. An optimal value function is more specifically the value function of an optimal policy. Value functions ...
Accepted

### Is value iteration stopped after one update of each state?

Where the author mentions the policy evaluation being stopped after one state, they are referring to the part of the algorithm that evaluates the policy -- the pseudocode you have listed is the ...
Accepted

### How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?

I don't understand how did we get rid of the condition $A_{t}=\pi'(s)$. We don't really, it is just moved into the subscript $\pi'$ in $\mathbb{E}_{\pi'}[]$ - it means the same thing here, that the ...
A policy can be stochastic or deterministic. A deterministic policy is a function of the form $\pi_{\text{deterministic}}: S \rightarrow A$, that is, a function from the set of states to the set of ...
I think this equation answer your question: $$q_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum_{a \in A}\pi^{i+1}(a|s)q_{\pi^{i}}(s,a)$$ value of the Q while taking ...