4
$\begingroup$

I often see the terms episode, trajectory and rollout to refer to basically the same thing, a list of (state, action, rewards). Are there any concrete differences between the terms or can they be used interchangeably?

In the following paragraphs, I'll summarize my current slightly vague understanding of the terms. Please point any inaccuracy or missing details in my definitions.

I think episode has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of $(s, a, r)$ sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.

With trajectory, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.

I think rollout is somewhere in between, since I commonly see it used to refer to a sampled sequence of $(s, a, r) $from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.

$\endgroup$
2
$\begingroup$

I don't really think there are fixed, different definitions for all those terms that everyone agrees upon. In most contexts they're going to be quite interchangeable, and if anyone is really using them in a context where they are supposed to have crucially important, different meanings, they should probably precisely define them right there.


I think episode has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of $(s,a,r)$ sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.

Agreed with this.

With trajectory, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.

I can't really think of cases where it's sensible to talk about trajectories with tuples shuffled into an arbitrary order. I'd still think of trajectories as having to be in the "correct" order in which they were experienced. But I do agree that trajectories can be little samples (for instance, little sequences of experience that we store in an experience replay buffer). So, every full episode would be a (long) trajectory, but not every trajectory is a full episode (a trajectory can just be a small part of an episode).

I think rollout is somewhere in between, since I commonly see it used to refer to a sampled sequence of $(s,a,r)$ from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.

I'd say that... often a rollout should have a "terminal" state as ending, but maybe not a true "initial" state of an episode as start. We might be in the middle of an episode, and then say that we "roll out", which to me implies that we keep going until the end of an episode. I don't think this term is as common as the other two in Reinforcement Learning, but more common in search / planning literature (in particular, Monte Caro Tree Search).

That said, when I'm working with MCTS I often like to put a limit on my rollouts where I cut them off if no terminal state was reached yet... so that isn't exactly a crisp definition either.

Due to how commonly-used this term is specifically in MCTS, and other Monte-Carlo-based algorithm, I also associate a greater degree of randomness with the term "rollout". When I hear "episode" or "trajectory", I can envision a highly sophisticated, "intelligent" policy being used to select actions, but when I hear "rollout" I am inclined to think of a greater degree of randomness being incorporated in the action selection (maybe uniformly random, or maybe with some cheap-to-compute, simple policy for biasing away from uniformity). Again, that's really just an association I have in my mind with the term, and not a crisp definition.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.