# If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?

Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41):

As an example, suppose there are several different k-armed bandit tasks, and that on each step you confront one of these chosen at random. Thus, the bandit task changes randomly from step to step. If the probabilities with which each task is selected for you do not change over time, this would appear as a single stationary k-armed bandit task, and you could use one of the methods described in this chapter.

If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?

## 1 Answer

The critical detail of the initial multi-k-armed bandit as it is described in the quoted text, is that the agent cannot detect or use knowledge of which bandit problem it is presented with on each time step.

Therefore the expected reward from action $$a$$ is the weighted sum of expected rewards from all bandits for that action. It is weighted according to the random selection, and stays fixed for the combined bandit, same as for the individual bandits that it is composed of. This is a single real value for expected reward, which means optimising it looks identical from the outside to optimising a simpler bandit with less "machinery".

All of this changes as soon as the agent is allowed to observe data, i.e. the context of which sub-distribution it is taking an action for in the next time step.

• This answer is not clear for me as well. I would be happy if it can be updated. Feb 12 at 19:35
• @DSPinfinity What exactly is not clear? It is pretty thorough IMO. I probably could not do better without prompting what the problem is. You may need someone else to answer Feb 12 at 19:36
• I think it will be clear if wording is changed and more explanations are added. Feb 13 at 7:07