I'm studying reinforcement learning now and I'm quite a newbie to this field. I have some questions about how to derive the equation as below.


The equation above is the probability of a trajectory.

But in my opinion, the equation should be, $p_{\theta}(s_{1},a_{1},\dots,s_T,a_T)=p(s_1)\prod_{t=1}^T\pi_\theta(a_t|s_t)\prod_{t=1}^{T-1}p(s_{t+1}|s_t,a_t)$

I don't know why $p(s_{T+1}|s_T,a_T)$ emerges in the original equation because there is no $s_{T+1},a_{T+1}$ in the trajectory. I think I'm missing something.

Can somebody help me?

  • $\begingroup$ This might be just bayesian factorization. Do you have the image of the trajectory graph? $\endgroup$ Aug 9 at 1:14
  • $\begingroup$ @desert_ranger Oh I don't have it. But it's just the general trajectory which starts at s_1 and then do the action a_1, so on and forth. I know it's a bayesian factorization, but I still don't know why s_T+1 comes in the first equation. $\endgroup$
    – Feel
    Aug 9 at 8:04


You must log in to answer this question.

Browse other questions tagged .