1
$\begingroup$

This question asks if there is a way to measure distance between policies that are in fact probability distributions.

In the case of continuous control with deterministic policies where they take a state as input and return an action vector, what would be the best method to measure how close two policies are from each other?

A naive approach that came to my mind would be to:

  • Run both policies A and B to produce a trajectory each and record all states visited.
  • For each state encountered by policy A, ask policy B which action it would take (and vice-versa). Hence we would have, for every state encountered, both A and B action vectors.
  • For each state, compare action vectors of A and B by using a common distance (Euclidean distance?)
  • Take the average (maybe maximum) of those distances.

Does it make any sense from a theoretical point of view?

Improve this question
$\endgroup$
1
  • $\begingroup$ Can you clarify what the action vector produced by the policy actually represents? If it's not a probability distribution, then why do you have an action vector? Is it a policy that produces multiple actions, all of them need to be taken at time step $t$? Is this a multi-agent system? Please, edit your post to clarify this. $\endgroup$
    – nbro
    Commented Dec 31, 2022 at 12:49

1 Answer 1

Reset to default
0
$\begingroup$

One way to measure the distance between two policies is to use the Kullback-Leibler (KL) divergence. This measures the amount of information lost when one policy is used to approximate the other.

Another way to measure is to use the Wasserstein metric. This measures the amount of "work" or "energy" required to transform one policy into the other.

A third way is to use the Mutual Information (MI) metric. This measures the amount of information shared between the two policies.

yes! Euclidean distance would way to measure the distance be between the two policies action vectors

It does, sort of. The idea is that you have a reward function that encodes the task you want to solve. The agent is trying to find an optimal policy that maximizes this reward function. In order to do this, the agent must learn the reward function.

One way to think about this is that the agent is trying to find a mapping from states to actions that will maximise the reward. This is the same as finding a mapping from states to a goal state that maximises the reward. So, if you have two different agents, they will both be trying to find a mapping from states to actions that maximises the reward. However, they will likely find different mappings, because they have different starting

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ OP’s question was regarding deterministic policies. I’m not sure how useful KLD is in this instance as they are trivially dissimilar (I think, happy to be told otherwise). $\endgroup$
    – David
    Commented Nov 16, 2022 at 11:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .