From what I learn from CS285 and OpenAI's spinning up, a trajectory in RL is a sequence of state-action pairs:

$$\tau = \{s_0, a_0, ..., s_t, a_t\}$$

And the resulting trajectory probability is:

$$ P(\tau \mid \pi)=\rho_{0}\left(s_{0}\right) \prod_{t=0}^{T-1} P\left(s_{t+1} \mid s_{t}, a_{t}\right) \pi\left(a_{t} \mid s_{t}\right) $$

  1. From CS285: http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf

  2. From spinning up: https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#trajectories

However, from my derivation, the above trajectory probability actually corresponds to the following sequence where the last action $a_t$ is absent:

$$ \tau = \{s_0, a_0, ..., s_t\} $$

Can someone please help me clarify this confusion?


1 Answer 1


If we use $T$ as the notation for the terminal state, then the last action is $a_{T-1}$. This is because when you reach state $s_T$ you don't take another action, which would be $a_T$, because the episode is finished upon reaching the terminal state.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .