2
$\begingroup$

As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies?

For example, IS (importance sampling)-based estimators, such as weighted IS or doubly robust, are still unbiased when they are used to evaluate UCB1, which is a non-stationary policy, as it chooses an action based on the history of rewards?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Weighted IS is not unbiased - it is initially biased towards evaluating the behaviour policy, and that bias is reduced as more samples are taken. So it cannot be "still unbiased" in other circumstances. Also, UCB1 as an example behaviour policy has other problems (such as how to define the policy in stochastic terms). In addition, what does it even mean to "evaluate a non-stationary policy"? By definition, the value changes. Please clarify: Is your question about evaluation or control scenarios (you would generally only use UCB1 in control), and is it specifically about UCB1? $\endgroup$
    – Neil Slater
    Commented Jun 21, 2020 at 9:32

0

Reset to default

You must log in to answer this question.