The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization.

The old policy $\mu$, or the behavior policy, is okay to be non-stationary in the IPS estimator even if the new policy $\nu$, or the target policy, should be stationary.

Is this true for the importance sampling (IS) estimator, which seems to be a variant of IPS, for off-policy evaluation in a reinforcement learning problem?

IS estimator is explained in this paper Doubly Robust Off-policy Value Evaluation for Reinforcement Learning.

The target policy should be stationary, but can the old policy be non-stationary in the IS estimator?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.