# Is the initialisation of $V(s)$ and $\pi(s)$ really arbitrary in policy iteration?

In Sutton and Barto's book (Reinforcement learning: An introduction. MIT press, 2018), the algorithm "Policy Iteration" is:

Here, $$V(s)$$ is initialized arbitrarily, meaning that I can choose anything I want. Moreover, I think nothing is stated about $$\gamma$$ here so we can consider undiscounted environments where $$\gamma = 1$$. Now suppose we use the following environment:

If I initialize:

• $$V(s_1) = 10$$
• $$V(s_2) = 10$$
• $$V(s_3) = 0$$
• $$\pi(s_1) = a_1$$
• $$\pi(s_2) = a_3$$

With this, it appears that the "Policy Evaluation" part will not have any effect and the algorithm will immediately stops, outputing a policy where the optimal actions are $$\pi(s_1) = a_1$$ and $$\pi(s_2) = a_3$$. What am I missing ?

EDIT: I made a toy repository to reproduce, if you want to tweak the numbers of point out something I misunderstood: https://github.com/Gregwar/policy_iteration_initialization/blob/master/policy_iteration.py

• (I think the thing is that $\gamma = 1$ can only be used if all episodes terminate, else it has to be $< 1$. Else the episode returns is not well-defined.) Feb 17 at 13:38

One simple rule to avoid this is to set $$\pi$$ to the stochastic equiprobable policy initially (i.e. the first policy will explore all edges of the MDP, making it very hard - maybe impossible - to construct bad cases), and all initial $$V$$ to $$0$$. Although these can be set arbitrarily, it is not the user of the solver that normally gets to decide these initialisations, but the person implementing the solver.
• I agree with your second point, but not with the first. If $\gamma < 1$ it seems to always work. I made a toy repository if you want to fiddle with the numbers and/or check if I did something wrong: github.com/Gregwar/policy_iteration_initialization Feb 19 at 11:06