On-policy distribution is defined as follows in Sutton and Barto:

enter image description here

On the other hand, state visitation frequency is defined as follows in Trust Region Policy Optimization:

$$\rho_{\pi}(s) = \sum_{t=0}^{T} \gamma^t P(s_t=s|\pi)$$

Question:: What is the difference between an on-policy distribution and state visitation frequency?

  • $\begingroup$ Where are you getting the definition of state visitation from, exactly (it's probably S&B again, but I'd like to know which chapter and section)? I am asking because the term is being defined for a purpose - also note unlike the distribution, its value can be above 1 for any given state. $\endgroup$ Dec 8, 2021 at 18:56
  • 1
    $\begingroup$ Definition of state visitation is from TRPO paper. $\endgroup$
    – user529295
    Dec 9, 2021 at 16:05


You must log in to answer this question.