4 votes
Accepted

Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for ...
nbro's user avatar
  • 40.1k
3 votes
Accepted

Why is Thompson Sampling considered a part of Reinforcement Learning?

Thompson Sampling (TS) is used in the context of bandits, which is a special case of the RL problem. You can also use TS for the full RL problem, but that can lead to inefficient exploration. To know ...
nbro's user avatar
  • 40.1k
3 votes

Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

Many techniques for the exploration/exploitation dilemma that are inspired by multi-armed bandit problems, such as UCB1, assume that you can explicitly enumerate all state-action pairs; in fact, multi-...
Dennis Soemers's user avatar
  • 10.2k
3 votes

Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem?

The first thing to note here is that your results seem aligned with the results commonly found in the bandit literature. Second thing to note would be that the performance of bandit algorithms is ...
user5093249's user avatar
2 votes

Should I use exploration strategy in Policy Gradient algorithms?

Neil Slater's answer is very nice, but I have a couple more suggestions: You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for ...
harwiltz's user avatar
  • 1,126
2 votes
Accepted

Should I use exploration strategy in Policy Gradient algorithms?

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state. Yes, ...
Neil Slater's user avatar
  • 31.5k
1 vote
Accepted

UCB, Thompson sampling etc seems myopic/greedy for bandits?

The (binary) multi-armed bandit actually is a MDP with one state and $K$ actions. So your suggestion boils down to meta-learning: Find the parameters of one MDP that can solve another. Let's go with ...
maxy's user avatar
  • 223
1 vote
Accepted

How to compute the action probabilities with Thompson sampling in deep Q-learning?

So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout? The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an ...
Neil Slater's user avatar
  • 31.5k

Only top scored, non community-wiki answers of a minimum length are eligible