3
$\begingroup$

When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We want to learn the target policy $\pi$ while behaving according to policy $\mu$. Therefore, for each transition $(s_t, a_t, r_t, s_{t+1})$, we apply the importance ratio $\frac{\pi(a_t | s_t)}{\mu(a_t | s_t)}$.

My question is: if we are training at every step, the behavior policy may change at each step and therefore the transitions of the trajectory $\tau$ are not obtained from the current behavior policy, but from $n$ behavior policies. Why do we use the current behavior policy in the importance sampling? Should each transition use the probability of the behavior policy of the timestep at which that transition was collected? For example by storing the likelihood $\mu_t(a_t | s_t)$ along with the transition?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and behaviour policy throughout the trajectory.

See the calculation below, where the product is happening for all the probabilities throughout the trajectories. (screenshot from Chapter 5, section 5.5 (page 85) of Sutton & Barto)

enter image description here

Improve this answer
$\endgroup$
1
  • $\begingroup$ I'm not sure whether this formulation from Sutton and Barto assumes that all experiences in the trajectory were generated with a behaviour policy when the value function was fixed (for all those experiences in the trajectory) or not. I think that's the crux of the question. If the value function changes while you're collecting experience for the trajectory, then the behaviour policy, which is derived from the value function, may also change (i.e. take different actions). $\endgroup$
    – nbro
    Nov 18, 2020 at 15:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .