In Policy Iteration (PI), the action generated by the policy, whether it's optimal or not w.r.t the current value function $v(s)$. Whereas, in Value Iteration, the action is greedily generated w.r.t current $v(s)$, which is an approximation of the objective function (as I understand). As a consequence, in the first few iterations, will Value Iteration perform better than Policy Iteration?

  • 1
    $\begingroup$ Is objective function the optimal value function $v^*$? $\endgroup$ – DuttaA Sep 11 at 17:31
  • $\begingroup$ @DuttaA That's what I'm trying to figure out. I'm new to reinforcement learing. I'm used to notations in optimal control and classic optimization, where objective function is used. Since reinforcement learning is solving an optimization problem, I assume the value function will eventually converge to the objective function (or the optimal value function)? Then the greedy action w.r.t the optimal value function corresponds to the optimal control action that minimizes the objective function in optimal control. $\endgroup$ – qiang li Sep 11 at 17:52
  • $\begingroup$ Well I don't know about classical control but in RL terminology is very important. Simple mistakes in terminology can lead to ambiguous interpretations (I also struggle a lot with RL terminology). But I think you should try to stick to exact RL terminology even if your intuition says you can name a particular thing better, because it usually leads to a different idea altogether. $\endgroup$ – DuttaA Sep 11 at 17:56
  • $\begingroup$ @DuttaA I totally agree. I'm still taking the first steps to figure out what everything means in RL and their corresponding counterparts in my problems. After that, I'll confidently formulate my problems with terminology the RL community agreed on. $\endgroup$ – qiang li Sep 11 at 18:10
  • $\begingroup$ @qiangli In general, an objective function is just the function you want to optimize (either minimise or maximise). It seems that you're using the expression objective function as a synonym for the function you want to learn or find. $\endgroup$ – nbro Sep 13 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.