0
$\begingroup$

Prediction's goal is to get an estimate of a performance of a policy given a specific state.

Control's goal is to improve the policy wrt. the prediction.

The alternation between the two is the basis of reinforcement learning algorithms.

In the paper “Safe and Efficient Off-Policy Reinforcement Learning.” (Munos, 2016), the section 3.1) "Policy evaluation" assumes that the target policy is fixed, while the section 3.2) "Control" extends to where the target policy is a sequence of policies improved by a sequence of increasingly greedy operations.

This suggests that even a proof of convergence is established with a fixed target policy, one cannot immediately imply that of the case where the target policy is a sequence of improving policies.

I wonder why it is the case. If an algorithm converges under a fixed target policy assumption, any policy during the chain of improvement should have no problem with this algorithm as well. With the merit of policy improvement, each policy in sequence is increasingly better hence converging to an optimal policy.

This should be obvious from the policy improvement perspective and should require no further proof at all?

$\endgroup$
  • $\begingroup$ I don't understand what you are asking. What do you mean by "This suggests that even a proof of convergence is established with a fixed target policy". Where does the suggestion come from? Also, you should rephrase this "one cannot immediately imply that of the case where the target policy is a sequence of improving policies.", which is not very clear. $\endgroup$ – nbro Sep 19 at 13:20
  • $\begingroup$ @nbro Actually, my simpler question is what is the need for section 3.2) "control" at all? $\endgroup$ – Phizaz Sep 19 at 13:42
  • $\begingroup$ @nbro About the suggestion, I imply from the existence of section 3.2. If the proof provided in the section 3.1 (fixed policy case) is suffice, there should be no section 3.2. $\endgroup$ – Phizaz Sep 19 at 13:43
1
$\begingroup$

It is a mathematical problem. Usually, we use contraction mapping theorem for the proof of convergence. You should apply the Banach fixed point theorem for Bellman's functions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.