1
$\begingroup$

Below is an example (p.89) from "RL and Optimal Control" book by D.Bertsekas on the construction of a a case study where the rollout algorithm is worse than the base heuristic on which the rollout algorithm is constructed.

It is not clear why sequential improvement is violated. First of all, the first sentence is not clear to me: is it the rollout algorithm or the base heuristic that accidentally produces an optimal control sequence?

It is not clear to me why sequential improvement is violated in the provided example. I will be happy for clear explanation.

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The first sentence simply asserts the base heuristic accidentally produces an optimal control sequence, otherwise the rollout algo starting from initial state $x_0$ would not be strictly inferior to the one produced by the base heuristic starting from the same $x_0$ as the author stated later.

This unwanted result of rolling out a strict inferior target policy under this section's deterministic problems assumption is due to violation of the base heuristic's sequential improvement property, which is a weaker property than sequential consistency shared by a deterministic policy, for example. Thus the base heuristic $H$ here must not be a deterministic policy such as the random-restart hill climbing search algo applied for the traveling salesman problem.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Your explanation is less clearer than the explanation in the book. $\endgroup$
    – DSPinfinity
    Mar 17 at 13:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .