As the title suggests, I have a doubt about the computation of the $Q_a$ used to update the delta and the $Q_a$ used to select the next action in the exploitation phase, as shown below (source of pseudocode in Figure 8.9).

Watkins' Q(λ) algorithm with linear approximation

In both for loops enumerated in the image with 1 and 2, the $Q_a$ are calculated considering the new state, but while in 1 for each action the active features $F_a$ are calculated, in 2 they aren't. So what are the active features to consider in 2? Are they the same as those in 1? In that case, I could avoid recalculating the $Q_a$ by storing those calculated in 1.



You must log in to answer this question.

Browse other questions tagged .