0
$\begingroup$

Sutton-Barto, page 164:

enter image description here

enter image description here

The main part of Figure 8.2 shows average learning curves from an experiment in which Dyna-Q agents were applied to the maze task. The initial action values were zero, the step-size parameter was $\alpha$ = 0.1, and the exploration parameter was $\epsilon$ = 0.1. When selecting greedily among actions, ties were broken randomly. The agents varied in the number of planning steps, n, they performed per real step. For each n, the curves show the number of steps taken by the agent to reach the goal in each episode, averaged over 30 repetitions of the experiment. In each repetition, the initial seed for the random number generator was held constant across algorithms. Because of this, the first episode was exactly the same (about 1700 steps) for all values of n, and its data are not shown in the figure. After the first episode, performance improved for all values of n, but much more rapidly for larger values. Recall that the n = 0 agent is a nonplanning agent, using only direct reinforcement learning (one-step tabular Q-learning).

My questions:

Q1- It is said that " In each repetition, the initial seed for the random number generator was held constant across algorithms".

Where in the Dyna-Q algorithm, random number generator used? Is it used in the $\epsilon-greedy$ (part b) or in the loop (f)?

Q2- It is also said that "Because of this, the first episode was exactly the same (about 1700 steps) for all values of n, and its data are not shown in the figure. After the first episode, performance improved for all values of n, but much more rapidly for larger values."

Dyna-Q is used in each step of an episode and hence it is not clear to me why first episode was exactly the same (about 1700 steps) for all values of n.

$\endgroup$
1
  • 1
    $\begingroup$ Please, do not write titles like "unclear point..." in the title. Just write your specific question in the title. $\endgroup$
    – nbro
    Commented May 10 at 13:02

1 Answer 1

1
$\begingroup$

Randomness is used in epsilon greedy (both for determining when exploration should happen and what action is taken) and by "Search Control" to pick a previous state-action pair to plan for. Planning will inevitably consume some of the random numbers so this explains "about 1700" and I think "exactly" refers to across repetitions for a given choice of n.

Remember, Search Control only selects from previously experienced state-action pairs. Before the first episode ends, no reward has been seen so sampling from previous pairs does not help in finding the solution. So, for all values of n planning steps the performance will be the same. Once the reward has been seen, in the second episode, planning can help propagate this reward back to the agent's present position.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .