3
$\begingroup$

I want to determine some distance between two policies $\pi_1 (a \mid s)$ and $\pi_2 (a \mid s)$, i.e. something like $\vert \vert \pi_1 (a \mid s) - \pi_2(a \mid s) \vert \vert$, where $\pi_i (a\mid s)$ is the vector $(\pi_i (a_1 \mid s), \dots, \pi_i(a_n \mid s))$. I am looking for a sensible notion for such a distance.

Are there some standard norms/metrics used in the literature for determining a distance between policies?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance are not necessarily metrics in a mathematical sense).

A common measure is the Kullback–Leibler divergence (which is not a metric, in a mathematical sense, given that it does not satisfy certain required conditions for being a metric). For example, in section 4 of the PPO paper, the KL divergence is used as a regulariser (which is actually quite common, for instance, in the context of variational Bayesian neural networks). The TRPO also uses the KL divergence.

The Wasserstein metric has also been used in RL, for instance, in distributional RL (but, in this case, not to compare policies but distributions over value functions).

You can find more info about statistical distances here. The specific distance that you use may depend on the problem that you want to solve and the properties that you want your distance to have. For example, the KL divergence is unbounded above, so, if that's not desirable, you could choose another one. The paper On choosing and bounding probability metrics (2002, by Gibbs and Su) may also be useful. Here I also talk about the KL divergence and total variation.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ Worth noting the Wasserstein metric requires there to be metric over the term that the distribution applies to, so may not apply well to discrete action spaces where there is no conceptual space or ordering between terms (e.g. what is the distance between "up" and "left" actions on a game console?) $\endgroup$
    – Neil Slater
    Dec 13, 2020 at 17:06
  • $\begingroup$ @NeilSlater Thanks for pointing this out. Right now, I am only familiar with the details of the KL divergence, although, as I say in this answer, I know that the Wasserstein metric has also been used in RL and I would guess that other measures of distance have been used. If you are familiar with the details of other measures (other than the KL divergence, which is quite popular), I think it may be worth providing another answer with another perspective, which maybe provides some guidance on how to choose it in the context of RL. $\endgroup$
    – nbro
    Dec 13, 2020 at 17:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .