What is a "trajectory" in reinforcement learning?

Question

I'm now learning about reinforcement learning, but I just found the word "trajectory" in this answer.

However, I'm not sure what it means. I read a few books on the Reinforcement Learning but none of them mentioned it. Usually these introductionary books mention agent, environment, action, policy, and reward, but not "trajectory".

So, what does it mean? According to this answer over Quora:

In reinforcement learning terminology, a trajectory $\tau$ is the path of the agent through the state space up until the horizon $H$. The goal of an on-policy algorithm is to maximize the expected reward of the agent over trajectories.

Does it mean that the "trajectory" is the total path from the current state the agent is in to the final state (terminal state) that the episode finishes at? Or is it something else? (I'm not sure what the "horizon" mean, either).

Answer 1

In answer that you linked, I may have used an informal definition of "trajectory", but essentially the same thing as the quote. A "trajectory" is the sequence of what has happened (in terms of state, action, reward) over a set of contiguous timestamps, from a single episode, or a single part of a continuous problem.

So $(s_3, a_3, r_4, s_4, a_4, r_5, s_5, a_5, r_6, s_6)$ taken from any scenario where an agent was used in the problem environment would be a trajectory - at least as I intended it in the answer. This could be from real-world data, or a simulation. It could involve a totally random or untrained agent, or a fully-optimised policy.

In the other definition that you have found, the focus on states and a horizon could make it slightly different, but actually I suspect that it is the same thing, as it is not that useful to only know the states. The Quora answer is probably just using "path of the agent through the state space" as shorthand to describe the same data.

A "horizon" in reinforcement learning is a future point relative to a time step, beyond which you do not care about reward (so you sum the rewards from time $t$ to $t+H$). Fixed horizons can be used as an alternative to a discount factor for limiting sums of reward in continuous problems. They may also be used in other approaches, but basically mean the same thing - a time step beyond which you don't account for what happens.

Answer 2

Neil’s answer is good but I’m observing that strictly translating the following Quora’s answer statement

In reinforcement learning terminology, a trajectory $\tau$ is the path of the agent through the state space up until the horizon

we get $\tau = \{ s_{t} \}_{t \in [t_{0}, t_{H}]} \quad s_{t} \in \mathcal{S}$ with

• $\mathcal{S}$ the State Space
• $t_{0}$ initial time
• $t_{H} > t_{0}$ time associated to a certain event $H$

So according to Quora's answer author, it should just be a temporal sequence of states (no action and rewards)

Answer 3

A trajectory ist just a sequence of states and actions. In RL, the goal is to maximize the reward, by finding the right trajectories.

$$ \operatorname{max}_\tau R(\tau) $$

This means maximizing not immediate reward (caused by one action from a state), but cumulative reward (all states and actions: trajectory)