My problem setting is that KL-divergence is between two policies based on the data of one trajectory. As two ways to be shown below:
- $KL(\theta_{1}, \theta_{2}) = \sum\limits_{k=1}^{N}\pi_{\theta_{1}}(a_{k}|s_{k})\log\left(\frac{\pi_{\theta_{1}(a_{k}|s_{k})}}{\pi_{\theta_{2}(a_{k}|s_{k})}}\right) $
- $KL(\theta_{1}, \theta_{2}) = \frac{1}{N}\sum\limits_{k=1}^{N}\sum\limits_{a \in \mathcal{A}}\pi_{\theta_{1}}(a|s_{k})\log\left(\frac{\pi_{\theta_{1}(a|s_{k})}}{\pi_{\theta_{2}(a|s_{k})}}\right) $
where N is the number of samples of a trajectory, and $|\mathcal{A}|$ is finite. The only difference is that the first method just uses the tuple $(s, a)$ of the trajectory, whereas the second method uses the distribution of state $s$, where averaging is considered.
So which method is correct or preferred? I'm almost not finding any useful information for this problem.
