0
$\begingroup$

My problem setting is that KL-divergence is between two policies based on the data of one trajectory. As two ways to be shown below:

  1. $KL(\theta_{1}, \theta_{2}) = \sum\limits_{k=1}^{N}\pi_{\theta_{1}}(a_{k}|s_{k})\log\left(\frac{\pi_{\theta_{1}(a_{k}|s_{k})}}{\pi_{\theta_{2}(a_{k}|s_{k})}}\right) $
  2. $KL(\theta_{1}, \theta_{2}) = \frac{1}{N}\sum\limits_{k=1}^{N}\sum\limits_{a \in \mathcal{A}}\pi_{\theta_{1}}(a|s_{k})\log\left(\frac{\pi_{\theta_{1}(a|s_{k})}}{\pi_{\theta_{2}(a|s_{k})}}\right) $

where N is the number of samples of a trajectory, and $|\mathcal{A}|$ is finite. The only difference is that the first method just uses the tuple $(s, a)$ of the trajectory, whereas the second method uses the distribution of state $s$, where averaging is considered.

So which method is correct or preferred? I'm almost not finding any useful information for this problem.

$\endgroup$

0

You must log in to answer this question.

Browse other questions tagged .