Skip to main content
1 of 2
pg2455
  • 221
  • 1
  • 5

On-policy distribution for Emphatic TD

The book by Sutton and Barto discussed in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M_t$, the emphasis, help in getting a state distribution closer to the target policy.

My understanding of on-policy distribution is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values). The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M_t$ helping in correcting for state distribution?

pg2455
  • 221
  • 1
  • 5