Skip to main content
added 8 characters in body; edited tags; edited title
Source Link
nbro
  • 41.4k
  • 12
  • 114
  • 205

On-policy distribution Why is $M_t$ (the emphasis) helping in correcting for the state distribution in the Emphatic TD algorithm?

The book by Sutton and Barto discusseddiscusses in section 11.8section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for the distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M_t$, the emphasisemphasis, help in getting a state distribution closer to the target policy.

My understanding of on-policy distributionon-policy distribution is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values). 

The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M_t$ helping in correcting for the state distribution?

On-policy distribution for Emphatic TD

The book by Sutton and Barto discussed in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M_t$, the emphasis, help in getting a state distribution closer to the target policy.

My understanding of on-policy distribution is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values). The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M_t$ helping in correcting for state distribution?

Why is $M_t$ (the emphasis) helping in correcting for the state distribution in the Emphatic TD algorithm?

The book by Sutton and Barto discusses in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for the distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M_t$, the emphasis, help in getting a state distribution closer to the target policy.

My understanding of on-policy distribution is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values). 

The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M_t$ helping in correcting for the state distribution?

Source Link
pg2455
  • 221
  • 1
  • 5

On-policy distribution for Emphatic TD

The book by Sutton and Barto discussed in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M_t$, the emphasis, help in getting a state distribution closer to the target policy.

My understanding of on-policy distribution is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values). The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M_t$ helping in correcting for state distribution?