I am trying to understand the formulation of the maximum entropy Inverse RL method by Brian Ziebart. Particularly, I am stuck on how to understand the computation of state - visitation frequencies.

In order to do so, they utilize a dynamic programming approach to compute the visitation frequency, in which the next state frequency is calculated based upon the state visitation frequency at the previous time step.

This is the algorithm below where, $D_{s_i,t}$ is the probability of state $s_i$ being visited at time step $t$.

enter image description here

What is the difference between this way of computing state visitation frequency compared to the naive method of summing the total number of times state $s_i$ appears in the trajectory divided by the trajectory length?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.