# Tag Info

Accepted

### What are the major differences between multi-armed bandits and the other well-known algorithms (DQN, A3C, PPO, etc)?

You should start with the general definition of Reinforcement Learning problem. And what Markov Decision Process is. DQN, A3C, PPO and REINFORCE are algorithms for solving reinforcement learning ...
• 1,972
Accepted

### Are bandits considered an RL approach?

Several important researchers distinguish between bandit problems and the general reinforcement learning problem. The book Reinforcement learning: an introduction by Sutton and Barto describes bandit ...
• 36.3k

### How to implement a contextual reinforcement learning model?

In the case in which some of the states -but not all- are fully independent of the actions -but still obviously determine the optimal actions-, how could we take these state variables into account? I ...
• 25.4k
Accepted

### How do I recognise a bandit problem?

The bandit problem has one state, in which you are allowed to choose one lever among $n$ levers to pull. Why is there just one state in the formulation of this bandit problem? There is one state ...
• 1,112
Accepted

### Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for ...
• 36.3k
Accepted

### Why is regret so defined in MABs?

In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew. The point of regret as a metric ...
• 25.4k
Accepted

### What is the probability of selecting the greedy action in a 0.5-greedy selection method for the 2-armed bandit problem?

I read section 2.2 of Sutton and Barto, and I understand your confusion: the $\epsilon$-greedy algorithm is not defined precisely on page 27-28. Selecting an action randomly "every once in awhile&...
• 1,112

### Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

Many techniques for the exploration/exploitation dilemma that are inspired by multi-armed bandit problems, such as UCB1, assume that you can explicitly enumerate all state-action pairs; in fact, multi-...
• 9,649

### Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem?

The first thing to note here is that your results seem aligned with the results commonly found in the bandit literature. Second thing to note would be that the performance of bandit algorithms is ...
• 722
Accepted

### Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?

Scale your neural network inputs. The raw observations are in range $[0,89]$, and neural networks will cope badly with that used as inputs. The ideal case for NN for each input feature is a gaussian ...
• 25.4k

### Can you convert a MDP problem to a Contextual Multi-Arm Bandits problem?

The main difference between an MDP and contextual bandit setting is time steps and state progression. If those are important to the problem you want to solve, then it is not possible to convert. ...
• 25.4k

• 25.4k
1 vote
Accepted

### Is there a way to form a reward function so that it would take into account the order of the actions?

You are describing an environment which requires a full Markov Decision Process (MDP) to model it and reinforcement learning (RL) algorithms to solve it. You will not be able to adapt k-armed bandit ...
• 25.4k
1 vote

### Is the Bandit Problem an MDP?

The bandit problem is an MDP. You can make the same argument about needing data to learn in the stateful MDP setting. The thing is, the data you need (the past rewards in this case) was drawn iid (...
• 1,061
1 vote

### Difference in UCB performance when scaling the rewards

Epsilon greedy is unaffected by scaling of rewards, it always selects a random action with a probability of epsilon. On the other hand, if we look at the formulation of UCB (Section 2.7 of ...
• 241
1 vote

### In UCB, is the actual upper bound an upper bound of an one-sided or two-sided confidence interval?

The upper bound used here is derived from Hoeffding's inequality, which provides a symmetric, two-sided confidence interval. A good pair of blog posts on how this bound used in UCB for bandits is ...
• 9,649
1 vote

### Multi Armed Bandits with large number of arms

Without any knowledge on the references you came across, I am assuming that the authors were considering common applications of MAB (planning, online learning, etc.) for which the time horizon is ...
• 196
1 vote

### When discounted MAB is useful?

One of the reasons a discount factor is used, is to make sure the reward maximization is a well-defined problem and to make the sum of all rewards convergent. In the MAB problem, the number of trials ...

Only top scored, non community-wiki answers of a minimum length are eligible