What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?
There is no difference. A policy decides which action to take in each state. This is usually split into deterministic policies of the form $\pi(s) : \mathcal{S} \rightarrow \mathcal{A}$ and stochastic policies of the form $\pi(s|a) : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}, \text{Pr}[A_t=a |S_t=s]$.
When we say that a policy is static, it means that the mapping from state to action - or distribution of actions - does not change over the time that we are interested in. This definition applies equally in the Reinforcement Learning (RL) and Contextual Bandit settings.
But in a reinforcement learning problem, the history can be used to define a state.
It can be used this way, but is not required. What is required is the Markov property i.e. that identifying the state also determines all the allowed transitions and rewards, plus their probabilities of occurring given an action.
In this case, does the history include the rewards revealed thus far as well?
If those can affect the future state transitions and rewards yes. If they generally do not, then you would usually exclude them from the state description.
If so, the policy is not stationary anymore I guess.
What you are proposing is a system that changes its state depending on rewards seen so far. As above, this is not necessary to define a RL problem, but it is allowed in RL. Whilst in a contextual bandit you assume no rules apply to state transitions, and this is a key difference between contextual bandit and RL settings.
The policy is stationary if its mapping rules remain unchanged.
Your proposed addition to the state does not require changing the policy. The policy is a function of the state. It may choose a different action depending on this historical aspect of the state, but it can still remain a fixed function - its input may change, but the policy function itself does not need to change to account for that.
