On-policy distribution is defined as follows in Sutton and Barto:

enter image description here

On the other hand, state visitation frequency is defined as follows in Trust Region Policy Optimization:

$$\rho_{\pi}(s) = \sum_{t=0}^{T} \gamma^t P(s_t=s|\pi)$$

Question:: What is the difference between an on-policy distribution and state visitation frequency?

  • $\begingroup$ Where are you getting the definition of state visitation from, exactly (it's probably S&B again, but I'd like to know which chapter and section)? I am asking because the term is being defined for a purpose - also note unlike the distribution, its value can be above 1 for any given state. $\endgroup$ Dec 8 '21 at 18:56
  • 1
    $\begingroup$ Definition of state visitation is from TRPO paper. $\endgroup$
    – user529295
    Dec 9 '21 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.