As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies?

For example, IS (importance sampling)-based estimators, such as weighted IS or doubly robust, are still unbiased when they are used to evaluate UCB1, which is a non-stationary policy, as it chooses an action based on the history of rewards?

  • 1
    $\begingroup$ Weighted IS is not unbiased - it is initially biased towards evaluating the behaviour policy, and that bias is reduced as more samples are taken. So it cannot be "still unbiased" in other circumstances. Also, UCB1 as an example behaviour policy has other problems (such as how to define the policy in stochastic terms). In addition, what does it even mean to "evaluate a non-stationary policy"? By definition, the value changes. Please clarify: Is your question about evaluation or control scenarios (you would generally only use UCB1 in control), and is it specifically about UCB1? $\endgroup$ – Neil Slater Jun 21 '20 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.