Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [contextual-bandits]

For questions about the contextual bandit (CB) problem and algorithms that solve it. The CB problem is a generalization of the (context-free) multi-armed bandit problem, where there is more than one situation (or state) and the optimal action to take in one state may be different than the optimal action to take in another state, but where the actions do not affect states (as e.g. in the reinforcement learning problem), but only the rewards.

Filter by
Sorted by
Tagged with
1 vote
0 answers
51 views

Python libraries for mulit-armed bandit problems [closed]

I am working on a problem that can be casted as a contextual bandit problem with continuous action space. I would like to tackle it by using something like the contextual zooming algorithm from the ...
Onil90's user avatar
  • 183
0 votes
0 answers
31 views

The impact of allowing the reward to be negative in a contextual bandit problem

I posted this question on stats.stackexchange.com 3 days ago. But there has been no answer. I would like to try my luck here. It seems the contextual bandits problems, as shown in these two papers ...
Hans's user avatar
  • 101
0 votes
0 answers
52 views

How to model the probability of click-through in bandit problem?

Consider a bandit problem in which we want to maximize the probability of click-through based on bid values ($b$ is the value of bid and $\Pr(b)$ shows the probability that a customer clicks on a link ...
Amin's user avatar
  • 481
1 vote
0 answers
64 views

Policy gradient (or more general, RL algorithms) for the problems where actions does not determine next state (next state is independent to action)

I am pretty new in RL. Could anyone suggest results/paper about whether or not policy gradient (or more general RL algorithms) can be applied to the problems where actions does not determine next ...
Penn's user avatar
  • 11
1 vote
1 answer
426 views

How to handled delayed rewards in contextual bandits [closed]

All the examples I see in the tf_Agents for contextual bandits, involves a reward function we generated the reward instantly after an observation has been generated. But, in my real world usecase (say ...
tjt's user avatar
  • 111
0 votes
1 answer
62 views

Does the policy search work if there is no state to state dependency through actions?

There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach ...
veerendra's user avatar
1 vote
0 answers
58 views

Multi-armed Bandit in optimization on graph edges selection

I have the problem, which I described below. I wonder if there exists a class of multi-armed bandit approaches that is related to it. I am working on computer networking optimization. In the simplest ...
Ramon's user avatar
  • 21
2 votes
0 answers
128 views

Is it better to model a Contextual Multi-Armed Bandit problem as an MDP with a non-zero discount factor than treating it as it is?

I'd like to ask if it is, generally, better to model a problem that naturally appears as a Contextual Multi-Armed Bandit like Recommender Systems as a Markov Decision Process with a non-zero discount ...
Daviiid's user avatar
  • 575
1 vote
1 answer
120 views

Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?

I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
Daviiid's user avatar
  • 575
1 vote
0 answers
531 views

(explore-exploit + supervised learning ) vs contextual bandits

Lets take an ad recommendation problem for 1 slot. Feedback is click/no click. I can solve this by contextual bandits. But I can also introduce exploration in supervised learning, I learn my model ...
dksahuji's user avatar
  • 111
1 vote
0 answers
169 views

What are the state-of-the-art learning algorithms for contextual bandits with stochastic rewards

I am building a solution for an environment with stochastic rewards in an online setting. I am wondering what the state of the art is in this setting. Is it $\epsilon$-greedy (with logistic regression)...
d56's user avatar
  • 243
8 votes
2 answers
3k views

What is the relation between the context in contextual bandits and the state in reinforcement learning?

Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)? Specifically, in RL, we can use a function approximator (e.g. ...
Maxim Volgin's user avatar
1 vote
0 answers
95 views

How do I learn the value function for a POMDP with a single-step horizon (bandit)?

Consider a POMDP with a finite number of environment states, $|\mathcal{S}| = N$, but the number of belief states is uncountably infinite. The belief state space is the convex hull of an $N$ simplex. ...
jdizzle's user avatar
  • 61
2 votes
0 answers
88 views

Is there a UCB type algorithm for linear stochastic bandit with lasso regression?

Why is there no upper confidence bound algorithm for linear stochastic bandits that uses lasso regression in the case that the regression parameters are sparse in the features? In particular, I don't ...
PJORR's user avatar
  • 21
5 votes
1 answer
1k views

Can you convert a MDP problem to a Contextual Multi-Arm Bandits problem?

I'm trying to get a better understanding of Multi-Arm Bandits, Contextual Multi-Arm Bandits and Markov Decision Process. Basically, Multi-Arm Bandits is a special case of Contextual Multi-Arm Bandits ...
peidaqi's user avatar
  • 151
3 votes
1 answer
2k views

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

I have a problem which I believe can be described as a contextual bandit. More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
gnikol's user avatar
  • 175
5 votes
2 answers
2k views

Are bandits considered an RL approach?

If a research paper uses multi-armed bandits (either in their standard or contextual form) to solve a particular task, can we say that they solved this task using a reinforcement learning approach? Or ...
user5093249's user avatar
1 vote
0 answers
73 views

How do I determine the optimal policy in a bandit problem with missing contexts?

Suppose I learn an optimal policy $\pi(a|c)$ for a contextual multi-armed bandit problem, where the context $c$ is a composite of multiple context variables $c = c_1, c_2, c_3$. For example, the ...
user31663's user avatar
3 votes
1 answer
686 views

How to implement a contextual reinforcement learning model?

In a reinforcement learning model, states depend on the previous actions chosen. In the case in which some of the states -but not all- are fully independent of the actions -but still obviously ...
freesoul's user avatar
  • 246
1 vote
1 answer
570 views

Why is it useful in some applications to use features that are shared by all arms?

In Li et al. (2010)'s highly cited paper, they talk about LinUCB with hybrid linear models in Section 3.2. They motivate this by saying In many applications including ours, it is helpful to use ...
wwl's user avatar
  • 153
1 vote
1 answer
351 views

How can I incorporate domain knowledge to choose actions in the case of large action spaces in multi-armed bandits?

Suppose one is using a multi-armed bandit, and one has relatively few "pulls" (i.e. timesteps) relative to the action set. For example, maybe there are 200 timesteps and 100 possible actions....
wwl's user avatar
  • 153
1 vote
0 answers
34 views

Name of a multiarmed bandit with only some levers available

In order to model a card game, as an exercise, I was thinking of an elementary setting as a multiarmed bandit, each lever being the distribution of expected rewards of a specific card. But, of course,...
arivero's user avatar
  • 51