From what I learn from CS285 and OpenAI's spinning up, a trajectory in RL is a sequence of state-action pairs:

$$\tau = \{s_0, a_0, ..., s_t, a_t\}$$

And the resulting trajectory probability is:

$$ P(\tau \mid \pi)=\rho_{0}\left(s_{0}\right) \prod_{t=0}^{T-1} P\left(s_{t+1} \mid s_{t}, a_{t}\right) \pi\left(a_{t} \mid s_{t}\right) $$

  1. From CS285: http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf

  2. From spinning up: https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#trajectories

However, from my derivation, the above trajectory probability actually corresponds to the following sequence where the last action $a_t$ is absent:

$$ \tau = \{s_0, a_0, ..., s_t\} $$

Can someone please help me clarify this confusion?


If we use $T$ as the notation for the terminal state, then the last action is $a_{T-1}$. This is because when you reach state $s_T$ you don't take another action, which would be $a_T$, because the episode is finished upon reaching the terminal state.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.