Prediction's goal is to get an estimate of a performance of a policy given a specific state.
Control's goal is to improve the policy wrt. the prediction.
The alternation between the two is the basis of reinforcement learning algorithms.
In the paper “Safe and Efficient Off-Policy Reinforcement Learning.” (Munos, 2016), the section 3.1) "Policy evaluation" assumes that the target policy is fixed, while the section 3.2) "Control" extends to where the target policy is a sequence of policies improved by a sequence of increasingly greedy operations.
This suggests that even a proof of convergence is established with a fixed target policy, one cannot immediately imply that of the case where the target policy is a sequence of improving policies.
I wonder why it is the case. If an algorithm converges under a fixed target policy assumption, any policy during the chain of improvement should have no problem with this algorithm as well. With the merit of policy improvement, each policy in sequence is increasingly better hence converging to an optimal policy.
This should be obvious from the policy improvement perspective and should require no further proof at all?
