During my machine learning lecture I saw the following statement on the slides: "Multi-armed-bandit selects an action based on the future reward"

Is that statement true?

In my opinion it is false because the selection of the action is not based on the reward directly as it is not known but on some strategy to maximize the reward in the long run.

Could maybe someone provide some references if it true or not, as I could not find any.


1 Answer 1


If you changed this:

"Multi-armed-bandit selects an action based on the future reward"

To this

"Multi-armed-bandit selects an action based on the expected future reward"

The description would be better in my opinion. Perhaps expected is just missing on the slide for brevity. I would rate the original quote as mostly true and give the author the benefit of the doubt here.

However, the quote is out of context, and how a multi-armed bandit agent selects an action can be more complex than simply considering the expected future reward. In bandit agents, the agent is usually considered to be learning "live", and has to balance exploring for better options versus exploiting a known best option so far. This is commonly referred to as minimising regret - which technically is a measure of how the agent performs in expectation (using actual action choices, but mean rewards for those actions as opposed to observed rewards) against an imaginary version of itself that always chose the best action in hindsight (or by revealing hidden knowledge of true values in the end for test cases).

If you include this training behaviour in the description, then you could also say:

"Multi-armed-bandit selects an action based on minimising regret"

Although this is not technically excluding basing on expected future reward, since usually the estimates of action values are used to make a decision, and "based on" is not the same as "only considering".


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .