In reinforcement learning, what guarantees that policy iteration would find the globally optimal solution and not just any local maximum?
I'm reading the book "Reinforcement Learning: An Introduction (second edition)" by Richard S. Sutton and Andrew G. Barto.
In chapter 4 they are discussing dynamic programming methods and on several occasions they mention that policy iteration is guaranteed to converge to the optimal policy because it satisfies the bellman optimality equation.
Bellman equation ... is a necessary condition for optimality
Necessary, but not sufficient. The bellman equation is non-linear so I don't see why there couldn't be multiple local maxima for the policy.
Why is policy iteration guaranteed to converge to the global optimum?