Newest 'multi-armed-bandit' Questions

0

votes

0answers

6 views

Is it better to model a Contextual Multi-Armed Bandit problem as an MDP with a non-zero discount factor than treating it as it is?

I'd like to ask if it is, generally, better to model a problem that naturally appears as a Contextual Multi-Armed Bandit like Recommender Systems as an Markov Decision Process with a non-zero discount ...

asked 2 days ago

Daviiid

4259 bronze badges

1

vote

1answer

52 views

Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?

I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...

reinforcement-learning q-learning dqn multi-armed-bandit contextual-bandit

asked Jul 8 at 11:45

Daviiid

4259 bronze badges

2

votes

1answer

37 views

Is the Bandit Problem an MDP?

I've read Sutton and Barto's introductory RL book. They define a policy as a mapping from states to probabilities of selecting each possible action. If the agent is following policy $\pi$ at time $t$, ...

reinforcement-learning comparison markov-decision-process notation multi-armed-bandit

asked Jun 15 at 8:44

Snowball

1835 bronze badges

3

votes

1answer

89 views

What is the probability of selecting the greedy action in a 0.5-greedy selection method for the 2-armed bandit problem?

I'm new to reinforcement learning and I'm going through Sutton and Barto. Exercise 2.1 states the following: In $\varepsilon$-greedy action selection, for the case of two actions and $\varepsilon=0.5$...

reinforcement-learning multi-armed-bandit epsilon-greedy-policy

asked May 23 at 20:48

Daviiid

4259 bronze badges

4

votes

1answer

91 views

What are the major differences between multi-armed bandits and the other well-known algorithms (DQN, A3C, PPO, etc)?

I have studied in the past different algorithms, i.e. DQN, DDQN, REINFORCE, A3C, PPO, TRPO, so on. I am doing an internship this summer where I have to use a multi-armed bandit (MAB). I am a bit ...

reinforcement-learning comparison deep-rl multi-armed-bandit

asked May 7 at 15:26

notaprogrammertoday

435 bronze badges

2

votes

1answer

66 views

Difference in UCB performance when scaling the rewards

I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...

reinforcement-learning rewards multi-armed-bandit upper-confidence-bound

asked Apr 23 at 15:55

d56

2131 silver badge6 bronze badges

1

vote

0answers

49 views

Which RL algorithm would be suitable for this multi-dimensional and continuous action space?

Is there an RL approach/algorithm that would be suited for the following kind of problem? There is a continuous action space with an action value $A_{a,t}$ for each action dimension $a$. The ...

reinforcement-learning multi-armed-bandit continuous-action-spaces algorithm-request

asked Feb 22 at 20:55

zaphod

112 bronze badges

1

vote

0answers

42 views

UCB-like algorithms: how do you compute the exploration bonus?

My question concerns Stochastic Combinatorial Multiarmed Bandits. More specifically, the algorithm called CombUCB1 presented in this paper. It is a UCB-like algorithm. Essentially, in each round of ...

multi-armed-bandit online-learning upper-confidence-bound

asked Feb 4 at 8:32

Hoeff

112 bronze badges

2

votes

1answer

67 views

In UCB, is the actual upper bound an upper bound of an one-sided or two-sided confidence interval?

I'm a bit confused about the visualization of the upper bound (following the notation of (c.f. Sutton & Barto (2018)) $$Q_t(a)+C\sqrt{\frac{\mathrm{ln}(t)}{N_t(a)}}$$ In many blog posts about the ...

machine-learning reinforcement-learning multi-armed-bandit upper-confidence-bound

asked Jan 19 at 20:23

D. B.

1014 bronze badges

1

vote

0answers

72 views

How do I learn the value function for a POMDP with a single-step horizon (bandit)?

Consider a POMDP with a finite number of environment states, $|\mathcal{S}| = N$, but the number of belief states is uncountably infinite. The belief state space is the convex hull of an $N$ simplex. ...

reinforcement-learning value-functions multi-armed-bandit pomdp contextual-bandit

asked Jan 14 at 14:54

jdizzle

613 bronze badges

5

votes

1answer

307 views

Multi Armed Bandits with large number of arms

I'm dealing with a (stochastic) Multi Armed Bandit (MAB) with a large number of arms. Consider a pizza machine that produces a pizza depending on an input $i$ (equivalent to an arm). The (finite) set ...

reinforcement-learning multi-armed-bandit epsilon-greedy-policy upper-confidence-bound

asked Dec 16 '20 at 16:03

D. B.

1014 bronze badges

1

vote

0answers

27 views

Multi-armed bandit problem without getting rewards

In a 2-armed-bandit problem, an agent has an opportunity to see n reward for each action. Now the agent should choose actions m times and maximize the expected reward in these m decisions. but it cant ...

reinforcement-learning multi-armed-bandit

asked Nov 23 '20 at 19:10

lighting

1111 bronze badge

2

votes

3answers

88 views

Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

Why aren't exploration techniques, such as UCB or Thompson sampling, typically used in bandit problems, used in full RL problems? Monte Carlo Tree Search may use the above-mentioned methods in its ...

reinforcement-learning multi-armed-bandit upper-confidence-bound thompson-sampling exploration-strategies

asked Nov 15 '20 at 19:10

Mika

1275 bronze badges

1

vote

0answers

31 views

Multi-armed bandits: reducing stochastic multi-armed bandits to bernoulli bandits

Agrawal and Goyal (http://proceedings.mlr.press/v23/agrawal12/agrawal12.pdf page 3) discussed how we can extend Thompson sampling for bernoulli bandits to Thompson sampling for stochastic bandits in ...

reinforcement-learning multi-armed-bandit thompson-sampling

asked Nov 8 '20 at 14:20

Felix P.

2771 silver badge6 bronze badges

3

votes

1answer

126 views

Why do we have two similar action selection strategies for UCB1?

In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit ...

reinforcement-learning policies sutton-barto multi-armed-bandit upper-confidence-bound

asked Oct 23 '20 at 10:32

nbro♦

28.1k6 gold badges56 silver badges122 bronze badges

3

votes

2answers

136 views

Why is regret so defined in MABs?

Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution. If we run the multi-armed ...

reinforcement-learning definitions rewards multi-armed-bandit regret

asked Sep 26 '20 at 15:45

epsilon-emperor

8901 silver badge13 bronze badges

3

votes

0answers

21 views

Mapping given probabilities to empirical probabilities

Consider following problem statement: You have given $n$ actions. You can perform any of them. Each action gives you success with some probability. The challenge is to perform given finite number of ...

reinforcement-learning math statistical-ai multi-armed-bandit

asked Sep 23 '20 at 21:48

Rnj

1815 bronze badges

2

votes

0answers

25 views

Is there a UCB type algorithm for linear stochastic bandit with lasso regression?

Why is there no upper confidence bound algorithm for linear stochastic bandits that uses lasso regression in the case that the regression parameters are sparse in the features? In particular, I don't ...

machine-learning reinforcement-learning linear-regression multi-armed-bandit contextual-bandit

asked Sep 20 '20 at 0:48

PJORR

212 bronze badges

1

vote

0answers

127 views

Understanding GLIE conditions for epsilon greedy approach

I was going through this course on reinforcement learning (the course has two lecture videos and corresponding slides) and I had a doubt. On slide 18 of this pdf, it states following condition for an ...

reinforcement-learning multi-armed-bandit epsilon-greedy-policy

asked Aug 28 '20 at 19:25

Maha

1312 bronze badges

4

votes

1answer

133 views

Can you convert a MDP problem to a Contextual Multi-Arm Bandits problem?

I'm trying to get a better understanding of Multi-Arm Bandits, Contextual Multi-Arm Bandits and Markov Decision Process. Basically, Multi-Arm Bandits is a special case of Contextual Multi-Arm Bandits ...

reinforcement-learning comparison markov-decision-process multi-armed-bandit contextual-bandit

asked Aug 17 '20 at 3:17

peidaqi

1411 bronze badge

1

vote

0answers

30 views

Customized food for persons based on their profile using Reinforcement learning

I am newbie to Reinforcement Learning, this is my idea - Agent(food provider) has to select a food based on the environment(based on the user profile). Here the reward will be given to the agent based ...

reinforcement-learning q-learning policy-gradients markov-decision-process multi-armed-bandit

asked Aug 13 '20 at 6:55

Bala u1

111 bronze badge

4

votes

2answers

113 views

Why do we use $X_{I_t,t}$ and $v_{I_t}$ to denote the reward received and the at time step $t$ and the distribution of the chosen arm $I_t$?

I'm doing some introductory research on classical (stochastic) MABs. However, I'm a little confused about the common notation (e.g. in the popular paper of Auer (2002) or Bubeck and Cesa-Bianchi (2012)...

papers notation multi-armed-bandit upper-confidence-bound

asked Jul 16 '20 at 13:41

MAB_N00B

413 bronze badges

1

vote

0answers

52 views

Solving multi-armed bandit problems with continuous action space

My problem has a single state and an infinite amount of actions on a certain interval (0,1). After quite some time of googling I found a few paper about an algorithm called zooming algorithm which can ...

reinforcement-learning multi-armed-bandit

asked Jul 9 '20 at 12:51

Peter

111 bronze badge

0

votes

1answer

421 views

Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem? [closed]

I ran a test using 3 strategies for multi-armed bandit: UCB, $\epsilon$-greedy, and Thompson sampling. The results for the rewards I got are as follows: Thompson sampling had the highest average ...

reinforcement-learning multi-armed-bandit thompson-sampling upper-confidence-bound epsilon-greedy-policy

asked Jun 15 '20 at 22:08

Java coder

1

5

votes

1answer

201 views

How do I recognise a bandit problem?

I'm having difficulty understanding the distinction between a bandit problem and a non-bandit problem. An example of the bandit problem is an agent playing $n$ slot machines with the goal of ...

reinforcement-learning terminology definitions multi-armed-bandit

asked Jun 8 '20 at 13:06

blue-sky

2691 silver badge9 bronze badges

1

vote

1answer

75 views

When discounted MAB is useful?

Many of multi-armed bandit(MAB) algorithms are used when the total reward is the sum of all rewards. However, in RL, the discounted reward is mainly used. Why is the discounted reward not prevailing ...

reinforcement-learning rewards multi-armed-bandit discount-factor

asked Jun 4 '20 at 21:16

Katatonia

4612 silver badges9 bronze badges

3

votes

2answers

203 views

Are bandits considered an RL approach?

If a research paper uses multi-armed bandits (either in their standard or contextual form) to solve a particular task, can we say that they solved this task using a reinforcement learning approach? Or ...

reinforcement-learning terminology multi-armed-bandit contextual-bandit

asked May 2 '20 at 14:42

user5093249

6622 silver badges7 bronze badges

1

vote

1answer

69 views

How do we reach at the formula for UCB action-selection in multi-armed bandit problem?

I came across the formula for Upper Confidence Bound Action Selection (while studying multi-armed bandit problem), which looks like: $$ A_t \dot{=} \operatorname{argmax}_a \left[ Q_t(a) + c \sqrt{ \...

reinforcement-learning multi-armed-bandit upper-confidence-bound

asked Apr 11 '20 at 6:42

SAGALPREET SINGH

1475 bronze badges

1

vote

0answers

37 views

How do I determine the optimal policy in a bandit problem with missing contexts?

Suppose I learn an optimal policy $\pi(a|c)$ for a contextual multi-armed bandit problem, where the context $c$ is a composite of multiple context variables $c = c_1, c_2, c_3$. For example, the ...

reinforcement-learning markov-decision-process multi-armed-bandit contextual-bandit

asked Nov 26 '19 at 21:54

user31663

111 bronze badge

2

votes

1answer

135 views

Solving a Multi-Armed, “Multi-Bandit” Problem

This is the problem: I have 66 slot-machines and for each of them I have 7 possible actions/arms to choose from. At each trial, I have to choose one of 7 actions for each and every one of the 66 slots....

reinforcement-learning markov-decision-process policy-gradients multi-armed-bandit

asked Oct 29 '19 at 1:23

FS93

1356 bronze badges

5

votes

1answer

244 views

It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?

I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space ...

reinforcement-learning policy-gradients multi-armed-bandit state-spaces continuous-action-spaces

asked Jun 12 '19 at 14:28

FS93

1356 bronze badges

2

votes

1answer

466 views

How to implement a contextual reinforcement learning model?

In a reinforcement learning model, states depend on the previous actions chosen. In the case in which some of the states -but not all- are fully independent of the actions -but still obviously ...

reinforcement-learning multi-armed-bandit contextual-bandit

asked Jun 12 '18 at 14:02

freesoul

2261 silver badge5 bronze badges

1

vote

1answer

292 views

How can I incorporate domain knowledge to choose actions in the case of large action spaces in multi-armed bandits?

Suppose one is using a multi-armed bandit, and one has relatively few "pulls" (i.e. timesteps) relative to the action set. For example, maybe there are 200 timesteps and 100 possible actions....

reinforcement-learning multi-armed-bandit action-spaces contextual-bandit

asked Apr 21 '18 at 13:51

wwl

1535 bronze badges

1

vote

0answers

32 views

Name of a multiarmed bandit with only some levers available

In order to model a card game, as an exercise, I was thinking of an elementary setting as a multiarmed bandit, each lever being the distribution of expected rewards of a specific card. But, of course,...

reinforcement-learning multi-armed-bandit contextual-bandit

asked Mar 2 '18 at 13:35

arivero

1112 bronze badges

2

votes

1answer

66 views

Programming a bandit to optimize donations

I'm developing a multi-armed bandit which learns the best information to display to persuade someone to donate to charity. Suppose I have treatments A, B, C, D (which are each one paragraph of text). ...

reinforcement-learning multi-armed-bandit

asked Feb 11 '18 at 16:03

wwl

1535 bronze badges

5

votes

1answer

220 views

What is a weighted average in a non-stationary k-armed bandit problem?

In the book Reinforcement Learning: An Introduction (page 25), by Richard S. Sutton and Andrew G. Barto, there is a discussion of the k-armed bandit problem, where the expected reward from the bandits ...

machine-learning reinforcement-learning terminology math multi-armed-bandit

asked Jan 18 '18 at 18:32

chessprogrammer

7093 silver badges15 bronze badges

Questions tagged [multi-armed-bandit]

Related Tags