# How to calculate KL-divergence?

My problem setting is that KL-divergence is between two policies based on the data of one trajectory. As two ways to be shown below:

1. $$KL(\theta_{1}, \theta_{2}) = \sum\limits_{k=1}^{N}\pi_{\theta_{1}}(a_{k}|s_{k})\log\left(\frac{\pi_{\theta_{1}(a_{k}|s_{k})}}{\pi_{\theta_{2}(a_{k}|s_{k})}}\right)$$
2. $$KL(\theta_{1}, \theta_{2}) = \frac{1}{N}\sum\limits_{k=1}^{N}\sum\limits_{a \in \mathcal{A}}\pi_{\theta_{1}}(a|s_{k})\log\left(\frac{\pi_{\theta_{1}(a|s_{k})}}{\pi_{\theta_{2}(a|s_{k})}}\right)$$

where N is the number of samples of a trajectory, and $$|\mathcal{A}|$$ is finite. The only difference is that the first method just uses the tuple $$(s, a)$$ of the trajectory, whereas the second method uses the distribution of state $$s$$, where averaging is considered.

So which method is correct or preferred? I'm almost not finding any useful information for this problem.