This question asks if there is a way to measure distance between policies that are in fact probability distributions.
In the case of continuous control with deterministic policies where they take a state as input and return an action vector, what would be the best method to measure how close two policies are from each other?
A naive approach that came to my mind would be to:
- Run both policies A and B to produce a trajectory each and record all states visited.
- For each state encountered by policy A, ask policy B which action it would take (and vice-versa). Hence we would have, for every state encountered, both A and B action vectors.
- For each state, compare action vectors of A and B by using a common distance (Euclidean distance?)
- Take the average (maybe maximum) of those distances.
Does it make any sense from a theoretical point of view?