My problem setting is that KL-divergence is between two policies based on the data of one trajectory. As two ways to be shown below:

  1. $KL(\theta_{1}, \theta_{2}) = \sum\limits_{k=1}^{N}\pi_{\theta_{1}}(a_{k}|s_{k})\log\left(\frac{\pi_{\theta_{1}(a_{k}|s_{k})}}{\pi_{\theta_{2}(a_{k}|s_{k})}}\right) $
  2. $KL(\theta_{1}, \theta_{2}) = \frac{1}{N}\sum\limits_{k=1}^{N}\sum\limits_{a \in \mathcal{A}}\pi_{\theta_{1}}(a|s_{k})\log\left(\frac{\pi_{\theta_{1}(a|s_{k})}}{\pi_{\theta_{2}(a|s_{k})}}\right) $

where N is the number of samples of a trajectory, and $|\mathcal{A}|$ is finite. The only difference is that the first method just uses the tuple $(s, a)$ of the trajectory, whereas the second method uses the distribution of state $s$, where averaging is considered.

So which method is correct or preferred? I'm almost not finding any useful information for this problem.



You must log in to answer this question.

Browse other questions tagged .