# What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?

A stationary policy is a function that maps a state to a probability distribution of actions.

In a contextual bandit problem, a state itself does not include the history. But in a reinforcement learning problem, the history can be used to define a state. In this case, does the history include the rewards revealed thus far as well? If so, the policy is not stationary anymore I guess.

As the title says, I think I am confused about the difference in the definitions of stationary (and/or non-stationary) policy between reinforcement learning and contextual bandit.

• A stationary policy is a policy that is fixed. So, it is not necessarily a function that maps a state to a probability distribution over actions (which is called a stochastic or non-deterministic policy), but it can map a state to a single action (a deterministic policy). So, both deterministic and stochastic policies can (or not) be stationary. – nbro Oct 3 '19 at 19:04
• What is your definition of a stationary policy in the context of the contextual bandit problem? – nbro Oct 3 '19 at 19:09
• @nbro as Neil defined in the answer below, if it is deterministic, a stationary policy is a function that maps a state (context in contextual bandit) to an action, and if it is stochastic, it maps a state to probability distribution of actions. – Hunnam Oct 4 '19 at 1:50
• Exactly. You just repeated what I said. – nbro Oct 4 '19 at 13:04

What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?

There is no difference. A policy decides which action to take in each state. This is usually split into deterministic policies of the form $$\pi(s) : \mathcal{S} \rightarrow \mathcal{A}$$ and stochastic policies of the form $$\pi(s|a) : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}, \text{Pr}[A_t=a |S_t=s]$$.

When we say that a policy is static, it means that the mapping from state to action - or distribution of actions - does not change over the time that we are interested in. This definition applies equally in the Reinforcement Learning (RL) and Contextual Bandit settings.

But in a reinforcement learning problem, the history can be used to define a state.

It can be used this way, but is not required. What is required is the Markov property i.e. that identifying the state also determines all the allowed transitions and rewards, plus their probabilities of occurring given an action.

In this case, does the history include the rewards revealed thus far as well?

If those can affect the future state transitions and rewards yes. If they generally do not, then you would usually exclude them from the state description.

If so, the policy is not stationary anymore I guess.

What you are proposing is a system that changes its state depending on rewards seen so far. As above, this is not necessary to define a RL problem, but it is allowed in RL. Whilst in a contextual bandit you assume no rules apply to state transitions, and this is a key difference between contextual bandit and RL settings.

The policy is stationary if its mapping rules remain unchanged.

Your proposed addition to the state does not require changing the policy. The policy is a function of the state. It may choose a different action depending on this historical aspect of the state, but it can still remain a fixed function - its input may change, but the policy function itself does not need to change to account for that.