The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization.
The old policy $\mu$, or the behavior policy, is okay to be non-stationary in the IPS estimator even if the new policy $\nu$, or the target policy, should be stationary.
Is this true for the importance sampling (IS) estimator, which seems to be a variant of IPS, for off-policy evaluation in a reinforcement learning problem?
IS estimator is explained in this paper Doubly Robust Off-policy Value Evaluation for Reinforcement Learning.
The target policy should be stationary, but can the old policy be non-stationary in the IS estimator?
