# What does "stationary" mean in the context of reinforcement learning?

I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy means that the policy does not depend on time, and only on state. But isn't that a unnecessary distinction? If the policy depends on time and not only on the state, then strictly speaking time should also be part of the state.

A stationary policy is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that the policy is not being updated by a learning algorithm.

If you are working with a stationary policy in reinforcement learning (RL), typically that is because you are trying to learn its value function. Many RL techniques - including Monte Carlo, Temporal Difference, Dynamic Programming - can be used to evaluate a given policy, as well as used to search for a better or optimal policy.

Stationary dynamics refers to the environment, and is an assumption that the rules of the environment do not change over time. The rules of the environment are often represented as an MDP model, which consists of all the state transition probabilities and reward distributions. Reinforcement learning algorithms that work online can usually cope and adjust policies to match non-stationary environments, provided the changes do not happen too often, or enough learning/exploring time is allowed between more radical changes. Most RL algorithms have at least some online component, it is also important to keep exploring non-optimal actions in environments with this trait (in order to spot when they may become optimal).

Stationary data is not a RL-specific term, but also relates to the need for an online algorithm, or at least plans for discarding older data and re-training existing models over time. You might have non-stationary data in any ML, including supervised learning - prediction problems that work with data about people and their behaviour often have this issue as population norms change over timescales of months and years.

• Comments are not for extended discussion; this conversation has been moved to chat.
– nbro
Mar 9 '20 at 15:40
• I had upvoted this answer, but now that I come back to it I'm wondering about the validity of your claim "It generally means that the policy is not being updated by a learning algorithm.". Can you please cite 1-2 papers or books (or whatever) where "stationary" is being used to indicate that the RL algorithm is not changing the policy? One could use that terminology, but I'm not sure I've ever seen it. I would also like to see something that supports this claim "Reinforcement learning algorithms that work online can usually cope and adjust policies to match non-stationary environments".
– nbro
Jan 21 at 15:12
• Moreover, when you say "Most RL algorithms have at least some online component", I mean most RL algorithms are online learning algorithms. Of course, now, with all the experience replays that may not be true anymore. In any case, that claim is also a bit misleading, in my opinion, because it seems that you're saying that most RL algorithms learn offline, but usually it's the opposite. Of course, I'm using the definition "online learning" as learning with one sample at a time as it arrives from some data-generating process (in this case, the environment and the interaction with it).
– nbro
Jan 21 at 15:14

A stationary policy is the one that does not depend on time. Meaning that the agent will take the same decision whenever certain conditions are met. This stationary policy may be probabilistic which implies that the probability of choosing an action remains the same. It may take different decisions but the probability remains the same.

A Stationary environment refers to the static model of the system. The model comprises of a Reward function and Transition probabilities. So, in a stationary environment, the reward function and transition probabilities remain constant or the changes are slow enough that the agent finds enough training time to learn the changes done in the environment.

You are right: a stationary policy is independent of time. It is basically a mapping from states to actions (or probability distributions over actions). Regardless of the point in time in which the agent observes the state $$s$$ it will select an action $$a$$ (or select a probability $$\pi(a \vert s)$$ for every action $$a$$).

There are two kinds of problem

## Stationary and non-stationary

Stationary problems are those whose reward value is static, dost not change and on other hand non-stationary problems are those whose reward value change with time