0
$\begingroup$

I would like to bring up a point regarding the application of on-policy algorithms, such as REINFORCE, to contextual bandit problems using data collected from other policies.

Here are my thoughts:

In a contextual bandit setting, the goal is to choose the action that maximizes the immediate reward given the current context. Unlike full reinforcement learning problems, there is no concern for future rewards or transitions between states.

My thought is that the focus on immediate rewards aligns well with the assumption that the policy used to collect the data may not play a role as in more complex RL settings involving state transitions and future rewards and thus, data collected from any policy can be used to update the policy being optimized.

However, on-policy algorithms like REINFORCE typically require data collected from the same policy that is being optimized. This ensures that the updates are directly related to the policy’s performance.

On the other hand, in REINFORCE, the policy is updated by making the "good" actions, the ones with the higher relative returns, more probable. In full RL context the returns are the sum of the cumulative rewards produced by following the policy that is being optimized. But, in contextual bandit context we only have immediate rewards and thus it is not necessary to use data collected by the policy being optimized since we do not have future rewards.

However, the primary concern I have with using off-policy data in on-policy algorithms is that the data may not accurately reflect the performance of the current policy being optimized and introduce potential biases and inaccuracies.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Say you have 2 deterministic slots, one that always returns 1 and the second that always returns 2 as rewards

Your behavioural policy selects the first slot machines 80% of the times, thus the second one the remaining 20% of the times.

Now, say you do 100 pulls, if you start with a uniform policy, you will do $1 \nabla \ln p(a_1)$ 80 times, and $2 \nabla \ln p(a_2)$ only 20 times, in other words, you get $80 \cdot (1 \nabla \ln p(a_1))$, and $20(2 \nabla \ln p(a_2))$ as reinforcement... I'm pretty sure that your policy will think that option 1 is the best

And that thinking that $a_1$ is the best, is only due to the bias that you are not correcting from the sampling

PSA: there exists the off-policy counterpart for REINFORCE that corrects for this bias (is always, and just, an importance sampling ratio correction as always)

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you very much Alberto for your answer. However, I think this is handled by substracting a baseline from the return (e.g., the average reward). That is the reason I mention "relative rewards" in my post. Thus, by substracting the average reward, the action with 1 reward will actually have negative relative return. $\endgroup$
    – gnikol
    Commented May 26 at 12:22
  • 1
    $\begingroup$ @gnikol that’s just a case, take 3 bandits, with deterministic reward 10,5,-50 and a behavioural with policy 5%, 80%, 15%, the baseline is 1 (if i’ve not messed up the math) and the most reinforced one is the second action by a large margin, and thus you will end up with such action as the best even in the off policy one $\endgroup$
    – Alberto
    Commented May 26 at 15:15
  • $\begingroup$ that makes sense. Thank you very much for your help. Except importance sampling are there any other methods to correct for this bias? $\endgroup$
    – gnikol
    Commented May 27 at 22:18
  • $\begingroup$ @gnikol yeah i also hate it, but never found alternative methods $\endgroup$
    – Alberto
    Commented May 28 at 9:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .