I am trying to understand the formulation of the maximum entropy Inverse RL method by Brian Ziebart. Particularly, I am stuck on how to understand the computation of state - visitation frequencies.

In order to do so, they utilize a dynamic programming approach to compute the visitation frequency, in which the next state frequency is calculated based upon the state visitation frequency at the previous time step.

This is the algorithm below where, $D_{s_i,t}$ is the probability of state $s_i$ being visited at time step $t$.

enter image description here

What is the difference between this way of computing state visitation frequency compared to the naive method of summing the total number of times state $s_i$ appears in the trajectory divided by the trajectory length?


1 Answer 1


Please look at line 5:

If $P(a_{i,j}|s_i)$ is equal to the policy that is used for generating the demonstrated trajectories, then it could be the same. However, in inverse RL you don't know $P(a_{i,j}|s_i)$ and you iteratively approximate it.

Furthermore, your technique would work if the MDP is deterministic or you would have an infinite amount of trajectories (central limit theorem). Otherwise, your summation neglects the probability of $s'$ given $s,a$ pairs, which is why you have $P(s_k|a_{i,j},s_i)$ in the equation.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .