2
$\begingroup$

The book by Sutton and Barto discusses in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for the distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M_t$, the emphasis, help in getting a state distribution closer to the target policy.

My understanding of on-policy distribution is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values).

The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M_t$ helping in correcting for the state distribution?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts.

An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning gives a proof of stability. At least parts of it should seem familiar if you've read Sutton and Barto's proof of the convergence of linear TD(0) on page 206 of their 2nd edition RL book.

On Convergence of Emphatic Temporal-Difference Learning gives a proof of convergence.

I confess that I don't understand these papers well enough to give a summary. If you eventually do, I would greatly appreciate an update.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi, after skimming through all cites of these two papers. I found it seems that there is no paper on how to combine Deep Networks with Emphatic TD. Is it possible to use it in DQN, TRPO or PPO? $\endgroup$
    – GoingMyWay
    Commented Jun 7, 2021 at 1:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .