1
$\begingroup$

Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41):

As an example, suppose there are several different k-armed bandit tasks, and that on each step you confront one of these chosen at random. Thus, the bandit task changes randomly from step to step. If the probabilities with which each task is selected for you do not change over time, this would appear as a single stationary k-armed bandit task, and you could use one of the methods described in this chapter.

Question: If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?

$\endgroup$

1 Answer 1

0
$\begingroup$

The critical detail of the initial multi-k-armed bandit as it is described in the quoted text, is that the agent cannot detect or use knowledge of which bandit problem it is presented with on each time step.

Therefore the expected reward from action $a$ is the weighted sum of expected rewards from all bandits for that action. It is weighted according to the random selection, and stays fixed for the combined bandit, same as for the individual bandits that it is composed of. This is a single real value for expected reward, which means optimising it looks identical from the outside to optimising a simpler bandit with less "machinery".

All of this changes as soon as the agent is allowed to observe data, i.e. the context of which sub-distribution it is taking an action for in the next time step.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .