# What does “stationary” mean in the context of reinforcement learning?

I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy means that the policy does not depend on time, and only on state. But isn't that a unnecessary distinction? If the policy depends on time and not only on the state, then strictly speaking time should also be part of the state.

A stationary policy is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that the policy is not being updated by a learning algorithm.

If you are working with a stationary policy in reinforcement learning (RL), typically that is because you are trying to learn its value function. Many RL techniques - including Monte Carlo, Temporal Difference, Dynamic Programming - can be used to evaluate a given policy, as well as used to search for a better or optimal policy.

Stationary dynamics refers to the environment, and is an assumption that the rules of the environment do not change over time. The rules of the environment are often represented as an MDP model, which consists of all the state transition probabilities and reward distributions. Reinforcement learning algorithms that work online can usually cope and adjust policies to match non-stationary environments, provided the changes do not happen too often, or enough learning/exploring time is allowed between more radical changes. Most RL algorithms have at least some online component, it is also important to keep exploring non-optimal actions in environments with this trait (in order to spot when they may become optimal).

Stationary data is not a RL-specific term, but also relates to the need for an online algorithm, or at least plans for discarding older data and re-training existing models over time. You might have non-stationary data in any ML, including supervised learning - prediction problems that work with data about people and their behaviour often have this issue as population norms change over timescales of months and years.

• "Stationary data" could maybe be associated with the responses of a stationary environment (in a model-free algorithm). In that case, assuming that the state transition probabilities and the rewards distributions are unknown, then the responses of the (stationary) environment to the actions of the agent could be considered "data" that the agent uses to build a model of the environment or to estimate a value function. This data might not be the same over time (because the environment might be stochastic), but the possible values (i.e. the distributions) do not change. – nbro Feb 15 at 11:50
• @nbro: In an optimal control scenario the policy changes, which makes the distribution of states, actions and rewards all non-stationary. The transition probabilities might not change (i.e. stationary dynamics), making any individual row equivalent over time if you just store $s, a, r, s'$, but the distribution of the data - which rows are likely to appear in the data - does in fact change. – Neil Slater Feb 15 at 12:31
• Why would the policy change in an "optimal control" scenario? Shouldn't it just change if the environment is non-stationary? What do you mean by "row"? You mean a trajectory of state, action, reward and next state? I think you're using an inappropriate terminology. The (probability) distribution, in statistics, is a function which tells you the probability of occurrence of certain values/outcomes. It seems like you're using the term "distribution of the data" to indicate the empirical data you obtain after having interacted with the environment. If not, then what do you mean by "data"? – nbro Feb 15 at 12:38
• @nbro: The policy changes because in a control scenario the goal is to find an optimal policy. You typically don't start with an optimal policy. I mean "row" in e.g. experience replay table, or any other way you would wish to view/arrange the data. Yes I meant "distribution of the data", but not just empirical measure. The state distribution $\rho(s)$ is a PDF, usually depends on the policy, and can have a large impact on function approximation approaches. I don't have space or time in comments to use terms so precisely - if you want to explore this in more depth please ask a new question. – Neil Slater Feb 15 at 12:44
• I always forget what we mean by "control". So, in the control context, the goal is to find the optimal policy, and, hopefully, the policy will change over time, until it's optimal. So, by "data", you meant the observed states, actions and rewards, which, of course, depend on the policy. – nbro Feb 15 at 12:47

You are right: a stationary policy is independent of time. It is basically a mapping from states to actions. Despite the point in time in which the agent observes the state s it will select an action a.

• Note that a stationary policy can still be a nondeterministic policy, so it doesn't have to always select the same action $a$ given the same state $s$. Its probability $\pi (a \vert s)$ of selecting $a$ given $s$ should remain fixed over time for it to be a stationary policy though. – Dennis Soemers Aug 20 '18 at 15:17